Today I want to share a tool that I have found indispensable for finding and fixing intermittent tests in test suites. It’s a little script I wrote, called ntimes.

Based on the commit logs to my dotfiles repository, until about 2014, to run the same command many times, I would press up in my terminal and press enter. While effective, this approach has the disadvantage of requiring me to be present at the machine and do manual work. I thought: there must be a better way.

So, probably by cribbing from somewhere and adding my own extensions, I made a script that could run an arbitrary command-line command multiple times and report a summary at the end. To use it, I would use something like:

$ ntimes 100 rspec spec/models/user_spec.rb:42

This would run that specific RSpec test or block one hundred times. At the end, the script prints how many times it succeeded and how many times it failed.

I also use Mac’s say command to get some audio feedback during test runs. It is a bit annoying to have it say “succeeded” whenever it successfully ran, but it can be useful to know when there was a failure (“guess I didn’t fix the issue…”). So I have it say a quick “failure” immediately when there is a failure (non-zero exit code of the command) and either “Success!” or “At least some failed…” at the end depending on the overall status. While it couples this script to Mac, you could probably extend it to use a more cross-platform approach.

Since it just operates based on the exit code, this command could be used on other programs to run them multiple times as well. If you don’t care about the successes or failures, you can still use it to run the same command multiple times.

Combining git bisect with ntimes as the command allows us to see when a test likely started being flaky. It is helpful to have a small scope of what could be causing the test to be intermittent. If it is because of a test setup issue in another directory, then you might have to run your entire test suite. (This would take much longer, so you might have to run it overnight.) ntimes is also quite helpful when taking over an existing code base that might have intermittent tests.

Sometimes if I’m worried about a test that I just wrote, I’ll do a proactive ntimes 100 on it just to be sure that I am not committing a test that will soon fail. I generally try to do this if I have really complicated before/after blocks or if I might be polluting global state.

To install ntimes on your machine, download it, make it executable, and then put it somewhere on your PATH. Please let me know if you found it useful!