Our Monolith Was Making Our Engineering Lag – Here’s How We Fixed It

Appsmith is a platform that helps developers build internal apps like support tools, dashboards, and asset-tracking apps – and it was built on a monolithic architecture from the very beginning. All of its components, from the front end to the back end, were in the same codebase. This approach had worked well for us while we were getting Appsmith off the ground, allowing for rapid innovation and iteration. It got us from product ideation to MVP, and all the way to a functional app development platform — but as our codebase and teams expanded, we began encountering problems.

Our monolith had become large, convoluted, and difficult to maintain. Instead of accomplishing more as the platform matured, we were accomplishing less. We were spending 90% of our time in maintenance or bugfix mode and could not realistically keep up with shipping new features. Seemingly straightforward changes like conditionally changing the color of rows in the table widget required months of work because different components were so interconnected when they should not have been. Our internal development processes were starting to generate drag.

This article is the second in a series about how we chose the right architecture for Appsmith. It discusses how we decided that a modular monolithic architecture — one that saw the deployment benefits of monolithic but allowed for easy collaboration through modularization — was the best path forward for our development teams and our community.

The problems with our existing architecture

The biggest problem with our existing codebase and development process was increasing confusion and cross-communication between teams about who owned which code due to lack of hierarchy and ownership. This also made getting new team members up to speed difficult. Due to the tangle of functionality and unclear responsibilities, we'd often see the following questions arise between teams:

• "Who will fix this bug?"
• “Does the responsibility for fixing the bug lie with the engineer/team that touched the file last?"
• “Is there a team with specific responsibility to build and maintain that product area?”

There was usually no clear answer to these questions — different components of the application were connected together in unintuitive ways. The straw that broke the proverbial camel’s back was when we added a new widget to the UI and it broke our REST API integration plugin. This should never have happened as the two should have nothing to do with each other!

Another problem was poor test coverage. We had approximately 5% test coverage, which was nowhere near enough to ensure a stable application. We knew that we had to improve this, but it was difficult to do so when the code for different functionalities was jumbled together.

We had run into Conway's law, which states that organizations design systems that mirror their own communication structure. Unrelated features breaking each other was the result of ad hoc and arbitrary communication between teams. And because different parts of the code did not have distinct jobs, it was hard to think in terms of inputs and outputs when formulating tests for different components.

At the same time as these problems were emerging, we were beginning our journey to open-source Appsmith; however, we knew that our architectural problems would make it difficult for contributors to work with us. Our codebase would first need revision and we had to decide: would we continue with our monolithic codebase, or split it up and take a microservices-based approach?

Keep our monolith or move to microservices?

Back when we were a startup, during the early phases of Appsmith’s development, a monolith made sense. However, we were not as organized as we could have been, and as development progressed we became increasingly aware that we would need to restructure our codebase. Exactly how we would restructure would be an impactful, lasting decision, so we had to consider the implications not just for our developers, but for end users who were trying to deploy the application and use it in production.

It had to solve platform development issues

Updating a mature platform's architecture is significant work. We had to present a case to ourselves for re-structuring our code: it had to be worth the investment and address the outstanding issues faced by our community, all without interrupting ongoing development and production use of the platform. We also needed our code to be easy to learn and work on to attract contributors and realize our OSS aspirations.

Once a project is up and running, it becomes increasingly harder to change its architecture. It is like changing an engine in a vehicle: easily done in the factory, but very hard while driving down the highway.

The chosen solution also had to work with the technologies that we were already using. For example, we use Yarn Workspaces to segregate our code, which requires all dependencies in a child folder to be added to the root folder, leading to increased maintenance to ensure that these libraries don't conflict with each other and are on the latest secure version.

This requirement makes modularity difficult. However, replacing Yarn Workspaces with an alternative such as Lerna would have necessitated making major breaking changes to our existing code, interfering with work being performed outside of the core Appsmith development team. It would be best if, where required, we could avoid imposing modularity where it would be especially disruptive until a suitable solution was devised.

These development considerations were a point in favor of the monolith. But we saw that if the monolith won out in the end, we would have to impose a strict hierarchy on both the codebase and ourselves as an organization.

It had to fit with our ethos of having the simplest possible deployment architecture

The bread and butter of Appsmith's business is our corporate clients, who demand on-premises deployments that are easy to install, update, and maintain. Reliability and longevity are key factors that business users will be looking at specifically when deciding to “buy in” to a platform.

Appsmith has invested significant resources researching and implementing what we think is the ideal deployment architecture, and any changes to the structure of our application could not undo this work.

Our single-container Docker binary also allows us to offer a single-click-to-deploy hosted version of Appsmith for users who want to try out the platform or aren't concerned about where it is hosted. Our architecture choice would have to lend itself to containerization in this manner — another point in favor of continuing with a monolithic architecture, provided that code quality would not suffer.

Any changes could not negatively impact the end-user experience

One of our biggest development considerations is deployment and user choice, as they make the greatest impact on end users’ first experience with our product. This was especially important if we were expecting an influx of new users from the open-source community. Any change couldn’t impact the core requirements surrounding this:

• Users must be able to decide where their data is stored and exactly which data is exposed to third-party services.
• Users must be able to choose which of Appsmith's bundled tools they use — for example, they may wish to use their own hosted MongoDB server rather than using the one we package in our binary.
• Users want the latest features and compatibility with the latest services and libraries so that they can pick which components are right for them.
• Users need to be able to build ”their way” — we want to eliminate boilerplate and provide flexible UI components to speed up development, not dictate how their apps should behave.

This functionality could be accomplished with both a monolithic architecture and a microservices-oriented one, so there was no clear winner here.

Incremental change is inherently better than extensive rewrites

We decided that a shift to microservices would add nothing to the end-user experience, at the cost of requiring extensive, potentially breaking, rewrites. Our monolithic architecture choice still made sense overall, but we still needed to figure out how to solve the problems that were piling up. The reason a microservices architecture looked attractive was because it imposes structure — we had a messy codebase and what we were really looking for was a pretext to tidy it up.

After this realization, the solution became apparent — we'd take a hybrid approach and modularize our monolith, retaining the advantages provided by our current architecture and augmenting it with the advantages provided by a microservices-based approach. We didn’t know it at the time, but we had independently stumbled onto what others had termed the modular monolith.

Realizing the development benefits of microservices in a monolithic architecture.

How we overhauled Appsmith’s architecture

Our development teams spent three weeks independently assessing the potential impact of keeping our existing monolithic structure compared with a potential shift to a modular monolith. Each team identified which parts of their codebase would need to change and decided whether it was feasible, carefully considering the possible effects on other parties.

We paid particular attention to the areas that produced the most bugs. They highlighted the systematic problems that either would be fixed by modularization itself, or could be fixed during refactoring. We also looked at which files were currently being modified, as that signified areas under active development. Finally, we examined the areas that would be affected by upcoming work according to our product roadmap.

Then we began to think very systematically about whether modularizing those portions was worth it through the lens of an investment of time and energy. There were many areas where it would have simply taken too long to overhaul the architecture and therefore wasn’t worth it (at least in the early stages). A strictly modular architecture would not have allowed us to make these compromises, to the detriment of development progress.

Once we were satisfied that we had made the right choices, we began overhauling Appsmith piece by piece. We started by updating the base infrastructure to make the architectural change possible, and now, as new features are added to Appsmith, they follow this modular approach.

Prioritizing what would benefit most from modularization

The area that would see the most benefit from refactoring was our UI widgets. Prior to modularization, creating a new widget took around six weeks, because the developer would have to make modifications to multiple files scattered across the codebase and then carefully test that nothing else had broken. We knew this was inefficient, and were confident that by containing all of the code and assets related to a widget in a single directory we could cut development time for a new widget to two weeks. With plans to add forty new UI widgets to Appsmith, prioritizing modularization of our UI components was obvious.

Another major area that we prioritized was the JavaScript evaluation engine. This provides autocomplete options to developers, lints their code for syntactical correctness, then executes the code and returns the result to be consumed by other parts of the application. Prior to modularization, changing certain functionality in this area affected other functionality in unexpected ways. For example, changing the linting logic actually slowed down code execution — another thing that just shouldn’t happen!

Once we modularized the JS evaluation engine, it became much easier for our developers to fix bugs, performance for code execution dramatically improved, and the number of serious bugs for the linter and autocomplete were reduced overall.

If we hadn’t put a pause on new features over the past year and prioritized refactoring these mission-critical components for modularity, it would have been impossible to add new features and make existing features more stable.

Evidence that we made the right choice

Due to our thorough consultation with our developers and users, and careful and methodical dissecting of our legacy code, we had full confidence in our plan. That said, I was elated to see the real-world results confirming that we made the right choice.

Improvements to our internal development processes

The fact that it was previously so easy to break functionality understandably caused developers to shy away from altering any nominally working code. Thanks to modularization, a rigid hierarchy of ownership and functionality means that developers can fix bugs without having to worry about something breaking in an unexpected place. Modularization has also increased productivity, reduced duplication of efforts, and all but eliminated instances where teams interfere with each other's work.

It also makes it easier to onboard engineers. We can teach new arrivals the “Appsmith way” of doing things and implement a clear ruleset on how they should be working on the codebase. With a better-organized code and organization structure we can take the SOLID software development best practices seriously, instead of just paying lip service to them.

###nAppsmith remains simple to deploy Keeping our deployment practices simple was very important to us. Our users’ install success rate went from 30% to 99% overnight after we switched to shipping a single Docker container, and our upgrade process is reliable as we don't have to concern ourselves with external dependencies or the host environment.

The modular monolith really came into its own here. A monolithic architecture for the underlying code aligns well with our single-container deployment architecture, while modularization encouraged us to clean up everything from the source code to the distributable binary as part of the process.

Our choice allowed our developers to continue working unfettered, while embracing open source

Open-source contributors tend to see monolithic applications as harder to work on compared to microservices-oriented projects, but I think that this is a misconception. Many projects are hard to work on not because of their architecture, but because they didn’t have any structure imposed on them from their early stages (often as a result of growing quickly, or because they began as someone's personal project), and because there is not adequate incentive to restructure them.

Just because monoliths don’t discourage disorganization doesn’t mean that they can’t be well organized. Modularization of a monolithic code base, whether from project inception or performed retroactively, addresses both confusing and difficult-to-maintain codebases and encourages better organization in development teams. While microservice-based development encourages, if not necessitates, heavy modularization, it is not exclusive to it.

Appsmith’s adoption of the modular monolith has made it easy to augment with new integrations and features — for example, our recent support for custom JS libraries. Our decision to refactor has also enabled our growing community of open-source contributors.

Ignore the popularity contests and decide what's best for your project

The current zeitgeist clearly favors splitting applications into microservices rather than containing everything in a monolithic codebase. I think that this is to the detriment of many small- to medium-sized development teams.

This doesn't make me a monolith zealot (it would be hypocritical, given my assertion that microservices are wrongly evangelized). Retaining the monolith could have potentially been the wrong choice — without consultation and planning, we could have made an entirely wrong decision for ourselves, for our users, or for our contributors. But, for Appsmith, a modular monolith ultimately turned out to be the right decision: from an end-user perspective, the only change has been an increase in reliability, while our developers have seen a 4x increase in the number of PRs merged, due almost entirely to a better architecture.

It's not about “monoliths vs. microservices,” it's about choosing what’s right for your project. I see many projects that adopt the microservices approach without even considering a monolithic architecture, rejecting it immediately as some relic of the past. However, for all but the largest of organizations, microservices can add considerable overhead to the development and deployment process.

My next article in this series, How Adopting a Modular Monolithic Architecture Enables Our Open-Source Development Process, will explore this further.

Cover artwork by Jemma Jose