Audit your test suite for speed

Fast tests are a key ingredient to staying focused. Instant feedback from your test suite gives you confidence that your changes haven’t inadvertently broken something and keeps you focused on the task at hand.

Over time as your application grows, tests that were once fast tend to get slower. It’s a gradual process, and you may not even notice the shift. It sneaks up on you until one day you realize a few tests are taking a long time. A really long time.

What now?

Audit your test suite to find its need for speed

In other words, find the slow spots and reorganize for your current needs.

We recently went through this exercise at ConvertKit. Our main application is a Rails app and we have the typical rails_helper file that’s included at the top of all tests. That file handles all the possible setup needed to run any kind of test we want to run:

• Cleans the database
• Seeds the database
• Configures database factories, redis, elasticsearch, cassandra, feature flippers, webmock, Sidekiq, custom rspec matchers, CI specific settings, etc.

And “rails_helper” includes 50+ other files too. These files and settings were added as needed over many years. They were all necessary for something at some point.However, they likely weren’t needed for every test and the overhead started to add up. Running a single test in our local development environment started to get slow.

How to get started auditing your test suite

With so many files and settings, it can be difficult to know what is necessary (or not) for any single test. Here’s how we did it.

Start with a copy of your “rails_helper” file, then remove as much as possible while a simple test still passes. You’ll learn a lot about your setup by doing this alone. Keep an eye on the speed of your test and make note of anything that added or removed significant time to the run.

Now that you have a basic test passing and a setup with the bare minimum to run it, start building back up and organizing around the needs of your tests. For example, many tests need database access so that is a good potential grouping. Or maybe your controller tests need a way to handle authorization. As you add these groups, inspect the configuration options of all parts and decide if they are still needed and have appropriate values.

Database access was our biggest speed gain from auditing our test suite

Database access turned out to be one of the biggest speed gains for our test suite. We found two scenarios that fell under “database access” that were working against each other when it came to speed.

In some cases, we wanted to start with an empty database, then seed it with data that is commonly needed in tests but that you don’t want to create anew for every single test. In other cases, we use transactional fixtures so any records created during the test are rolled back at the end.

So if we were rolling all of it back, did we need to start with an empty database? For us the answer was no, so we made these two different setup options.

More details about this speed gain within our setup

We “clean” our test database with truncation. This means that for each run of the test suite, all the tables are dropped, then recreated. With nearly 300 tables, this takes a while even on a fast computer. We use rspec and if you run a single test it is considered the full “suite” so each run of a single file was doing all of that work even when it wasn’t necessary. By making this an optional part of the test setup we are able to skip all of that work and speed up a single test run dramatically. We also found gains in our Elasticsearch index setup and seeding.

Below are examples of running a single test file with “rails_helper” and “test_setup” to show how much time was saved in some cases. All of these were run using Spring.

Using “rails_helper” with a test that accesses the database.

Finished in 17.76 seconds (files took 5.26 seconds to load)
14 examples, 0 failures

The same test as above, using “test_setup”.

Finished in 1.47 seconds (files took 0.37922 seconds to load)
14 examples, 0 failures

The same test again, but with the tests that access the database skipped.

Finished in 15.69 seconds (files took 5.12 seconds to load)
14 examples, 0 failures, 3 pending

And with “test_setup” also with the database tests skipped. These kinds of tests saw the biggest speed boost.

Finished in 0.09185 seconds (files took 0.47639 seconds to load)
14 examples, 0 failures, 3 pending

The nice thing about the “rails_helper” file that included everything that you could possibly need was that it was the same single line in all test files. That’s about as easy as it gets. Having each test file declare what it needs is more code, but not so much more that it’s cumbersome. It could also help identify where slowdowns are coming from in the future. Below is an example of how we do it when we need FactoryBot for creating ActiveRecord records.

require "test_setup"
TestSetup.require!(:factories)

Speed up your development workflow by auditing your test suite for speed

Even if you don’t reorganize your test set up by groups it’s worth auditing what you have and which parts are the slowest every so often. Take everything out until you have a bare bones version. Then build it back up bit by bit, making note of anything you may not need any longer. You’ll learn a lot about how things are configured and maybe even get some nice speed gains.

We’d love to hear what kind of improvements you made to your application’s test suite!