How Opinionated Design Helps Decrease Complexity
Good systems take away unnecessary choices. The fewer, the better.
Nobody loves to argue all day due to disparate opinions.
It is because it drains your energy, and you get wrong—a lot.
However, to design a compelling system that withstands the lifetime of a business, opinions, and principles must be enforced in all of its developer cultures.
Last week, the product wanted to onboard a couple of events into the system, and we will need to change this service to allow these events. This system filters allow-list events to reduce the amount of unnecessary events received downstream.
This system has evolved to become a very complex monster codebase - with wildcard and regex within the pattern matching that causes adding and removing allow-lists events nearly impossible.
An engineer submitted a PR by putting the prefix of the event as a wildcard inside of regex.
I pushed back on the PR, requested a change request, and provided my opinion.
"It is best to maintain a restrictive approach in this system to decrease the complexity of the filter function. From now on, we should avoid using wildcards and regular expressions. If stakeholders wish to introduce a new event, they must clearly specify which event they intend to onboard."
The engineer agreed with my statement. However, he thought it was too restrictive and tedious to update the repo for one more event added that conforms to those prefixes of the event. Further, he said that if we are required to add exact events to filter everything, it will be hard to maintain going forward.
I can empathize with him because I used to think the same. However, this service has been very complex and hard to maintain in the first place because every engineer who makes changes in this repository doesn't have guidelines. Moreover, they want to be "flexible" and welcome a flood of other "nice to have" events into the system that are conducive to more complicated systems.
The root cause that evoked that discussion was that the service started without strict and opinionated guidelines. Thus, with un-opinionated guidelines, we will end up with more complex systems that are not maintainable.
The Truth About Complexity
An ideal software stack usually involves some framework, shared business logic, some custom logic, and the main function, which, in Functional programming, we call the end of the world. For instance, we will have a framework, be it Akka, Finagle, or Typelevel stack, and then, on top of that, we will have shared business logic. This logic is often shared across multiple services, for instance, the model or a client wrapper around the internal price engine service. Then, there is the custom logic, which is logic that each one of your services is writing on, and last is the main flow function that you are writing your application with.
However, in reality, you will have something like this:
The custom logic layer usually involves multiple similar functions but does different things. As your code evolves, there will also be duplication code and code that no one knows if it is being used elsewhere.
What does all this software stack context have to do with complexity?
Each layer of the stack has an inherent complexity, and you will have to pay that complexity somewhere down the line if you don't address it in this layer. The minimum complexity must be addressed on each layer to reduce the complexity further down the pipeline.
Someone must set opinionated principles on how the team should design the system to elevate complexity. And that principles need to be reinforced in each decision of the design.
This principle is similar to Amazon's leadership principles. Bill Carr mentioned in the Book Working Backward, "Good intentions don't work. Mechanisms do." That means people already had good intentions when the problems cropped up in the first place. Therefore, Amazon realized early on that if you don't enforce these opinionated decisions early in the process, it will create problems and make them recur.
In building a good system design culture in your team, you must address these principles to avoid creating more complexity.
Fewer Opinions incur More Debt
It may be common to hear that engineers are lazy. While laziness can be a positive trait for programmers and engineers in certain contexts, it is not always desirable when creating the core of a large system. When engineers make fewer decisions, others are forced to make more decisions, which can lead to inconsistent outcomes. This lack of consistency can result in divergent or duplicated decisions. Developers working within your system may lack guidance, which is not their fault. As a result, they may do whatever they can to get the job done.
Let's put the vague statement above into concrete examples. You are doing a task that requires getting items from the database with some filter. You saw that a client has already created a get item from the database. However, that function doesn't contain additional parameters on the filter tag. Many people will rather overload the function with an additional parameter instead of doing surgery on the original function by including an additional filter parameter. Sooner or later, if there is no enforcement or set of guided frameworks from the repository owner, there will be a different overloaded get function that does the same thing but with a slightly different twist of the filter parameter.
What happens if there are no opinionated principles enforced in the codebase?
Engineers will start putting their own flavor in the code. The complexity is pushed to maintainers and the implementer of the system, and some engineers in the future will need to pay for the complexity (a.k .a. harder troubleshooting issues, longer developer timeline, larger story point estimation, etc.)
That is why engineer loves Frameworks!
Because framework helps you create strict guidelines and decrease the number of choices you can make. The thing about React. It heavily uses the functional programming paradigm. Relentlessly preaching to its users that mutation is a crime. Therefore, developers follow its rules without having their own flavor create some mutation in their state. Consequently, creating a simple and maintainable system.
Good Systems are Opinionated System
Good systems take away unnecessary choices. The fewer, the better.
My last article talks about best practices in API design. One of the tips was to pass in ONLY to define necessary attributes in the API. This is a deliberate and opinionated choice. The fewer attributes a system receives, the less complexity it creates. Having other attributes and fields that is "nice to have" can create a magnitude of other unexpected IO operation because that system now has access to those "nice to have" data.
Being Opinionated is Hard
Being the one who enforces strict guidelines is hard because no one wants to be a developer who prevents others from doing things.
Emphasizing the word "unnecessary" can be challenging. Being right and not being wrong are two different things. Deciding not to choose will not make you wrong, but it does not mean you are making the right decision. It is a way of avoiding responsibility. It is important to acknowledge that mistakes are inevitable, and we may not always get it right. When in doubt, it is better to omit information, as adding things later is easier than removing them.
General Guidelines on Creating an Opinionated Systems
Make it Simple
That doesn't mean it's easy. Making things simple is usually more complicated.
However, if you want to make your system easy, you must restrict many things and give them clear guidelines regarding success in operating the system.
You cannot be wrong for not doing anything. However, that doesn't mean that you are right. Providing guidance helps people find an easier way to implement an additional feature to their system.
Being Okay with Iteration
Creating a robust system that can withstand time requires constant iteration. That means you have to be okay with the system evolving to accommodate the current state of the business.
For instance, if you promote immutability in all of your system codebases, there is a time when mutability is preferred to improve its performance. In those scenarios, you should be flexible and okay with refactoring some parts of the codebase to accommodate those guidelines.
Retrieved Clear Guidelines
Many of our problems begin with the need for more clarity around requirements. The problem with the example at the beginning of the article is that the initial engineer who created the system needed to clarify the definition of required values and how those values will be used. Moreover, they were lazy enough to want to change the codebase for every onboarded event.
Consequently, the unclear guidelines create complexity and blurry roles around each stack of the codebase, pushing the problem further to future system implementers.
Recap
The complexity of software systems is often an outcome of unclear guidelines and the absence of enforced principles. Without opinionated decisions to guide development, engineers tend to make ad-hoc choices, leading to a proliferation of redundant code, lack of consistency, and increased maintenance challenges.
The discussion highlighted the need for clear, opinionated principles in software design. These principles act as mechanisms to prevent unnecessary choices, reduce complexity, and make the system more maintainable. While being opinionated can be challenging, it is essential for creating robust and easy-to-maintain systems. A key aspect of this approach involves:
Clearly defining necessary attributes and guidelines.
Promoting simplicity.
Being open to iterative changes based on evolving business needs.
Ultimately, embracing opinionated systems and enforcing clear guidelines not only simplifies development but also ensures a more structured, maintainable, and adaptable software architecture, thereby enhancing the overall efficiency of the development process.