How We Reduced Our Annual Software Expenses by $80,000 Using Appsmith

Many businesses from a huge variety of industries rely on the Appsmith low-code app platform to power their day-to-day operations — so it needs to be rock solid. Updates that add features or improve ease of use absolutely cannot break the apps our customers build. We test everything incredibly thoroughly.

As Appsmith continues to grow, so does the expense to our organization for the software and infrastructure required to test each change we make to our product. Reducing the number of tests was out of the question (we need to test everything!), so we decided to replace some of the increasingly costly tools that we use in our testing process with an Appsmith application — using our own product to solve our own in-house problems.

This article explains how we did this and the monetary and resource savings we achieved. Businesses in any industry can apply these processes to improve their workflows and reduce costs by replacing their legacy tools.

The problem: existing tools do not scale well, are not flexible, and are costly

My role at Appsmith is quality assurance. I make sure that the software that reaches our users is of the absolute highest quality,  so if a change that breaks functionality for our users were to slip through... well, I would have some tough questions to answer.

Pivotal to our quality assurance process is our test suite: the software that tests each and every update we make to Appsmith. If a bug is introduced, or someone makes a change to functionality that conflicts with another change, the test suite automatically flags it to the responsible party so that they can fix it and submit it for re-testing. This test suite is constantly expanding with the scope and pace of Appsmith development — more developers are contributing more features to our platform, and more contributions = more tests.

We use the Cypress testing tool for this task. We currently have over 2500 tests in our suite, each of which automatically runs every time we make a change to the Appsmith code. We use 60 GitHub runners to execute these tests in parallel. These runners — short-lived containers that execute computing tasks — come at a cost, but are necessary to make sure that the tests are run as quickly and efficiently as possible.

The core of Cypress is open source and free to use, while additional features like test parallelization and report generation (both of which we relied on) are paid enterprise functionality. The pricing model for accessing these features is usage-based, so as the quantity of our tests increased so did our costs. Our Cypress enterprise bill had eclipsed $80,000 per year — we needed to find an alternative, and fast.

The solution: use Appsmith to manage the development of Appsmith

As existing open-source and commercial alternatives did not meet our requirements, we decided to build our own solution. We used Cypress plugins to implement our own test parallelization and export the results to a SQL database and Appsmith to build a custom reporting dashboard to interact with this data.

Cypress organizes its tests into files called spec files. The first plugin that we built for this project allocates eligible specs for the GitHub runners before Cypress begins its execution, splitting them equally in terms of time required so that they finish at roughly the same time. At the current load of 2500 tests, run on 60 parallel runners, every run takes about 50–60 minutes.

The second plugin takes the results of these tests and writes them to a database. Even though our purpose was to reduce our reliance on Cypress's enterprise features, Cypress itself is still the best tool for the job, especially as we can extend it using plugins and inspect its output using hooks. We validate each test as it completes. On a failure, Cypress automatically records the test artifacts such as screenshots and videos of the problem that was encountered and uploads them.

If you're technically inclined, you can check out the code for the first and second plugins.

The final component that ties this all together is the dashboard — a graphical user interface created using Appsmith. We used list, button group and chart widgets to quickly build out the user interface and implemented any remaining bespoke functionality using JS Objects. We implemented access control using Appsmith's granular access control.

Creating this reporting tool — from conception, to design, to building the plugins and dashboard, and testing it — took two developers three weeks. For those not familiar with software development, this is a very fast turnaround for a custom application of this scope. This was possible because we were building on the foundations of the Appsmith platform rather than coding everything from scratch.

The results: a scalable testing platform that works the way we do and eliminates all of the costs

Alongside the significant cost savings, this was a highly rewarding project for us as it really shows off how impactful Appsmith can be for an organization. Businesses in any category can look at these results and investigate which of their own expensive software tools could be replaced to improve productivity and reduce costs.

Our new testing dashboard saves our engineers time and lets them focus on quality

Below you can see the final result — our Cypress plugin has run the tests in parallel and reported the results to the Appsmith dashboard. For each test we can see what it was, what triggered it, and who is responsible for it, as well as its final status using the color-coded button group.

Each test run shown in the dashboard corresponds to a change made to the Appsmith codebase and summarizes: the developer responsible (1), the repository affected (2), what triggered the test run (3), and its current status (4). For the trigger, "push" happens when a developer merges their changed code into the Appsmith development code repository and "repository_dispatch" happens when the developer wants to merge their code into the main codebase. The status is color-coded, with green indicating that a test run passed, red that it failed, dark gray that it was skipped, and light gray showing pending test runs.

Users can filter and search the results, and they can click on a test to see its test failure rate graph. From there, clicking on a failure (red bar) reveals more detail about the failure and its cause.

Failure rate graph for a test.

That detail includes the testing artifacts uploaded by the Cypress plugin, including screenshots and recordings of the failure occurring, as well as technical details and log output.

Details, logs, and artifacts from a failed test.

Everything our development teams need to quickly react to failed tests is available in this dashboard. This greatly reduces the amount of effort required to track and fix bugs and the amount of friction between remote teams collaborating on tasks.

Additional benefits to owning our own toolchain

There have been some additional benefits to this project, too. Not only have we saved money and streamlined our testing process, now that we're in full control of our tools in this area, we are able to add functionality that didn't exist in the proprietary solutions we previously relied on. We’re also free of the effects of vendor lock-in such as unexpected price increases and functionality changes that counter our workflows.

If we want a custom report, or we see a feature in another product that we think will improve our workflow, we can simply implement it in our own dashboard. Our tools can match our desired processes exactly.

You can use Appsmith to build your own dashboards and management tools too

In this case, we've used Appsmith to solve a cost problem, and also wound up improving our internal processes as a result. Our platform isn't limited to building software testing dashboards (it'd be a bit niche if it was) — it can be used to build any kind of software interface for your employees and customers. Appsmith is designed to be flexible and is used in diverse industries from fintech to human resources and even agriculture.

Appsmith can retrieve, display, and update data from any number of tools all in one place. You can automate processes, speed up data entry, and build incident response and customer support interfaces that bring together data from different platforms like Shopify, Airtable, Zendesk, and almost any other source.

You can get up and running with Appsmith in minutes for free on our cloud-hosted platform. If you have custom needs for your business, contact our team to discuss how Appsmith can meet your unique requirements.