Discover more from Edward's Newsletter
The Unspoken Rule Of Designing a Successful Systems
Deriving a solution doesn't have to be an Engineering Solution.
📣 Announcement
Don’t miss out on last week’s article:
Over the past six years of working in technology as a software engineer, it is not about the spoken rules that matter but the unspoken ones.
The unspoken rules are certain ways of doing things beyond your job description. The rules help you become a strong engineer and get promoted.
These are the rules that high-performers often follow without they didn't even realizing.
You can say that these rules are tribal knowledge, but these rules were discussed behind the coffee chat or mentor session, and they have been passed down through multiple stages of one's software engineering career.
Today, I will give you some patterns you should know to execute products and design systems successfully. Without further ado, let's go!
Know Your Short/Mid-term/Long-term Decisions
There is no such thing as perfect design. The number one rule of system design interviews was to check if the candidates could understand all kinds of tradeoffs.
Knowing how to create short-term, mid-term, and long-term solutions to a problem with the tradeoffs in between can help create a better decision process for the team.
It's important to understand how long your system will live and design them differently based on the duration of your system.
Is this system temporary?
Where do we need them for a seasonal flash sale and won't use them afterward?
Will this system be used later on across other verticals?
Asking product will result in, “I want it NOW, and it won't go away”.
Therefore, great product engineers will have the current solution, which is a quick-n-dirty, and the long-term solution, which will require some resources and time.
Engineers usually over-complicate systems. This is also because of the influences of blog posts on system design - system design videos and case studies usually design the system in an ideal state scenario.
Design pattern works when you work for a critical feature in Meta or Google that will receive millions or even billions of traffic. The best engineers usually design their system to be best suited to the current context of the business needs and extend or expand the system as needed. They are creating a system that can streamline the iteration process without sacrificing user experience.
I also get bogged down by over-designing the system, and one helpful tip is to start with short-term solutions.
These solutions will usually be the first that quickly pops into your head when asked the questions. Then, take extra time to develop several short-term solutions to ensure that your approach can lay the path toward the wanted future.
Lastly, context matters when designing short, mid-term, and long-term solutions. Understanding the importance of this problem regarding your team and your organization. Is it the user experience in the app? Is it minimizing the maintenance effort for another team? Is it to increase the process of customer service answering questions to the user?
Think about Metrics and Monitoring Early on In the Process
We think about metrics and monitoring at the end of the process.
Metrics and monitoring should be included early on in the design process.
One of the reasons is that it gives engineers the ability to:
Think in terms of business problems.
Think about maintenance
Testing helps confirm that your function works. Monitoring helps confirm that your product works.
One tip about this is to understand what metrics you want to gather. There are two types:
Business Metrics
Engineering Metrics
Business metrics come with understanding the action in features that move the needle. For instance, YouTube used to think that subscriber count is the core metric that will move the needle. However, view count and share are better metrics for the platform's growth. Medium initially thought that claps and view were their core metrics for engagement. However, reading is a better metric for evaluating the article's engagement. Knowing these business metrics can help you understand how to gather the information and data. You won't have a list of ad-hoc metrics sent in your application.
Engineering metrics are metrics that disclose your system works properly. A JVM metric can help visualize how often your system reboots or stops operating if you work with JVM languages. When a garbage collector kicks in, the system will halt, which causes the restart. Having such metrics helps you understand how frequent GC pauses occur and failure handling.
Monitoring also helped create a compass for what needs to be changed and iterated upon. Therefore, adding them early on will be invaluable.
Solution Doesn't Have to be an Engineering Solutions
Your solution doesn’t have to be through code.
Although code does solve a lot of people's problems.
Learn to question various solutions and consider whether the no-code solution is possible.
One great example is when David Pan, a former VP of MixPanel, mentioned this engineer who can "engineer out of a problem."
His team was looking to optimize our infrastructure spending by reducing our expenses on the Google Cloud Platform (GCP) and assigning a lead engineer to manage this project.
Many software engineers view this as a software problem, focusing on making the software more efficient or migrating databases to achieve greater efficiency. However, the lead engineer we have in mind sees this as a customer problem.
He hypothesizes that by analyzing the top ten customers in Mixpanel, we can identify the ones who contribute disproportionately to our infrastructure costs. By digging deeper into the top 10 cost generators for the company, we can identify those where the economy is upside down, i.e., we are spending more on them than they are paying us.
The lead engineer is confident he can convert this complex software engineering problem into a customer issue. He will find a way to incentivize these top customers and significantly reduce our infrastructure costs at Mixpanel.
On the other hand, another example of solving human problems with software is a good case study of Uber's former Senior Staff Engineer Matt Ranney, who talked about his lessons learned from scaling Uber to 2000 Engineers, 1000 Services, and 8000 repositories in October 2016.
He said, "Microservices are a way to replace human communication with API coordination." It's easier for teams to write code rather than having people talking and dealing with team politics. Aside from system load problems with microservice, the biggest main driver of adopting microservice is allowing teams to form quickly and run independently. As a hyper-growth company like Uber in 2016, the team needs to be formed quickly to work on something where they can reason about the boundaries. Then, they can also be responsible for the code they write each time.
Opinionated Design is Better Than Non-Opinionated Design
A good design is opinionated.
Why?
Because it takes away unnecessary choices.
Avoiding decisions does not guarantee being right, nor does it mean being wrong. It simply means not choosing and abandoning responsibility, which is not a successful strategy. Making mistakes is inevitable, and it is important to acknowledge and learn from them.
This happens in IOS vs. Android:
Android is more lenient in its design. On the one hand, hundreds of different phone service companies use Android as their operating system. Thus, each company has its design patterns. On the other hand, only 1 company uses the IOS operating system, Apple.
iPhone can be arguably more popular than Android because of its genius marketing campaign and because it gives you a framework for using its operating system (closed system instead of open). You, as a user, have fewer things to care about and more things to just use that design philosophy.
I have written more in-depth about opinion design in this article. If you want to read them below.
I was creating a scala library at Disney Streaming Service that powered millions of streaming users worldwide. When I had it written and submitted for feedback, some people mentioned that the library is tight-knit to Cats, the Scala-type level library, and is less flexible.
While tightly coupled with a specific paradigm or library may sound like a bad choice, the opinionated design helped simplify the number of choices required to do the job. Any new developer that uses the service and touches the library doesn't need to decide whether to use functional or object-oriented programming. From the start, we have clarified that this service will use a functional programming paradigm because it helps us scale the parallelism of millions of transactions.
Leverage Existing Stack When Possible
Imagine if you are tasks to design an index for search and researching for databases.
You found that Elastic Search is a better tool for the job. But none of your team is using Elastic Search.
They use Postgres right now.
You know that Elastic Search is a better tool for the job. Should you use them?
Again, every choice that you make will incur consequences.
What is the feature within ElasticSearch that you need that doesn't exist in Postgres? By choosing ElasticSearch, you will need to create a new infrastructure, get a security review, and understand all the nuances that may occur since ElasticSearch is a new data storage that has yet to be done in your team. On the other hand, Postgres has been running in production for well over the years, and there has been some gotcha about Postgres that your team has experienced.
How about the learning curve? How quickly can you onboard your other team members to operate with ElasticSearch? You will also require robust documentation so that new engineers can come and pick up the work when you are on vacation.
You will always need to weigh between the cost of learning vs painting a new system, and those costs will also cascade to the cost of engineering and financial costs.
Have a tendency to always question if a new technology is worth the complexity.
If the advantages can overcome the disadvantages and that having such technology can increase the productivity or speed of the system and bring $ $ $ of engagement to your feature. It may be a good way to switch.
Recap
Throughout my tenure as a software engineer, it's all those little tips and tricks you won't find in your job description, from making smart design choices to thinking ahead about metrics and not overcomplicating solutions.
This article guides you on some of the best practices for designing a successful product.
So, what are some of the unspoken rules you experienced during your career? Drop your tips in the comment below!
