Don’t Build Things From Scratch. ‘Plumb’ Your Code Instead.
When to Invent When Not To Invent When Building Good Software
I have a confession to make.
Early on in my career, I was diagnosed with the Not Invented Here Syndrome.
This disease appears in me and many of my fellow software engineers, who tend to reject suitable external solutions to software development problems in favor of internally developed solutions.
I remember spending numerous sprints trying to convince my team to create our cache layer and my retry function from scratch - thinking that this retry and caching layer would be different and better than the ones outside.
The realization hits when a staff engineer is compelled not to reinvent the solution but to reuse and leverage the existing technology.
And he stopped me with a simple ask, "If you made it from scratch, you will be responsible for maintaining them fully."
I felt embarrassed at first, but it is required for me to get out of this syndrome.
Through the years, I have learned these hard lessons and have adopted to look first for an existing solution and only reluctantly create something internally once thoroughly convinced that no existing solution can be reused or adapted.
Certainly, as we have more open source and existing tools to use out of the box, the need to build in-house is minimal.
The primary problem with such a mindset for developers would be the hero mindset - where they would rather be known to develop a framework that relies on the whole company instead of the guy who suggests this 3rd party.
Writing code is more satisfying than reading code.
On the other hand, this is also caused by the culture where output is more rewarded by outcome.
If you ask someone in any profession how to solve a problem, they will try to solve the problem on the side that they are good at. Therefore, most software engineers are always thinking of ways of writing more code to solve problems.
In this article, I wanted to share tips about when to leverage existing tools and when to invent in-house.
When to Leverage Existing Tools
The answer is almost all the time!
Travis Kalanick, the ex-Uber CEO, once said, "Product is slow, but ops is fast."
It takes time for engineers to solve a problem; the founders could have solved it faster manually. Thus, founders should utilize engineering only when they're overwhelmed with operations.
, the author of refactoring, wrote in one of his articles about choosing technology. He recommends making asymmetric bets on choosing a well-proven and robust technology for the core and critical of your business and exploring more exciting technology on the new services.He even firmly believes that engineering work should be about patching things together rather than building them from scratch - he called this Fermi's paradox of building.
One caveat in leveraging existing free and open-source technology is that we treat it like a free puppet. When you get a free puppet, take them to the vet since they may have some disease you never know. Like open-source software, you must be mindful and evaluate them to ensure they are properly used for your needs.
If you need to choose between an internal library and an open-source library, you should choose an internal library. This is because internal libraries usually have the following:
It is constructed to fit into your technology stack, such as monitoring, metrics, and logging.
have been approved and battle-tested in the production environment and have been approved by security
You don't need to take out more engineering resources to maintain; you won't be blamed if something breaks because of that internal tool.
The last point was half joking :)
However, reusing an existing library or a programming language technology decreases the mental context switch you need.
This includes setting up your client, creating your dependency injection, and structuring your files in your repository.
Consistency helps in this regard, making it a breeze to troubleshoot the codebase and understand how the system works if all the microservices within the company are set up consistently.
Michael Bryzek highlighted in InfoQ in 2018 that all APIs and services contain consistent naming in Flow.io.
"That consistency is key because we want to feel like all of our service is built by only one person for all of our users using our API, whether it is internal or our client."
One of the takeaways from this quote is that there are other ways for the most valuable resources in your company to innovate, and creating a new library from scratch is not one of them. In other words, reusing makes you develop features faster.
If you are given a task to implement a system, such as a caching layer or a Database ORM, the best way to check is to see if one has already been created. If there is already one created and it has been there for a while, the codebase has been battle-tested. It has gone through the production environment, is used by millions, and is stable.
In addition, existing tools usually provide community support. Be it an internal tool used by multiple services or an open-source library. There is community support that helps you answer questions and updates. If you want to choose existing tools, having a large community support prevails ones without.
I am not innovative if everything is just a rough patch and reuse.
Using existing tools doesn't mean sacrificing innovation. On the contrary, it often frees up resources, allowing teams to focus on creating unique features and functionalities that differentiate their product.
When to Build In-House
When it comes to building in-house, a couple of considerations :
Does this solve our custom problems?
Does inventing this solution give us a competitive edge?
While the upfront investment in custom tools can be higher, they can offer significant long-term benefits regarding efficiency, scalability, and competitive edge.
Clement Mihailescu mentioned in one of his videos that when building Algo Expert, an interview prep interview platform, it creates its own in-house remote execution functionality because it is very expensive and slow. In addition, they want to ensure that they have a competitive edge on their software and do not rely on the core functionality created by another third party.
Tesla developed their in-house software to provide over-the-air firmware updates, enabling remote improvements in safety, performance, and infotainment capabilities. These over-the-air updates reduced carbon emissions and saved time and money for consumers. As a result, Tesla saw a significant increase in vehicle deliveries.
The key takeaway is that, in some cases where you don't want to reinvent the wheel. However, sometimes, it will be smarter to reinvent the wheel based on the financial and product perspective. Most of those core features can give you competitive advantages and intellectual property.
Best Practices and Strategies
When thinking of building vs leveraging existing technology, a good caliber is to think more like an investor.
If I build this from scratch, how long will it be? Then, calculate that value with your hourly wages, and think, "Is there a better use of my time in solving something else that is more impactful?"
Solving more impactful problems is a better use of your time as an engineer, and when you put that in your impact log, it will be much more impactful than building a new library from scratch.
Another consideration for building vs. leveraging existing technology is to understand the following:
if the feature that you are building is a core feature
the current phase of your product
If you are building an MVP or a prototype and the future of the product is bleak, you better reuse the existing technology because you want to increase the ship-to-market rate quickly.
If you are in the growth phase where scalability and reliability are required, you can evaluate to see if building your own is required. Make sure that your pain point is significant enough to warrant this approach.
I recommend always reusing the existing tools before building them to save time on maintaining and ensuring they are rapidly battle-tested. There is a saying that Old technology is sharks, not dinosaurs because they have stuck around and survived rapid changes that constantly occur in the technology field. Therefore, if you want to swing your bets and make a home run, don't bet against old technology. Replace them only if you have a very good reason.
These tools won't be flashy and exciting, but they will help you and your team have a great night's sleep every week.
Conclusion
The journey from a 'Not Invented Here' mindset to embracing external solutions reflects a maturation in the software engineering approach. While crafting custom solutions can be enticing, often driven by a desire for innovation or a hero mindset, the practicality and efficiency of leveraging existing tools cannot be overstated. This approach accelerates development and allows engineers to focus on truly impactful work differentiating their product. As we navigate the vast software development landscape, the key lies in striking a balance—knowing when to build in-house for competitive advantage and harnessing the power of existing, battle-tested technologies.
In what scenario would you rather build your tools in-house?
Loved it. For a long time I've been thinking about the opposite advice: do reinvent the wheel (from time to time). I'll try to put my thoughts together and write it once and for all.