When developing things with a large suite of unit tests or automated specifications, inevitably tests that actually pass fail for some reason. More difficult is the case where the test or spec fails only intermittently. I’ve taken the approach of late to keep a file around that records when I run into a problematic spec. When I get to five times that the spec failed, it’s time to refactor it to be less troublesome.

The first thing I check is to look at the spec and the code it executes and ensure that I think nothing is actually broken. Next, I ensure that the spec is actually adding value. If it’s slightly broken and is actually useless, we might as well get rid of it now and not spend more time on it. Generally though, the functionality of a sporadically failing test will be correct, and the spec will add some value.

At this point, I run the test enough times to see that it fails consistently, or where it fails. With rspec, I set up spork and configure rspec to look for an external drb server. If you want clarification of this process, please leave a comment. I’m just trying to get this out of my head at the moment.

spork allows me to run the file with the problematic spec much more quickly. I’m talking several orders of magnitude. So then, you can run the spec via your shell in a loop to get the results. The snippet below runs the troublesome spec fifty times, and appends each run to a file.

$ for i in {1..50}
      rspec spec/models/troublesome_model_spec.rb >> temp.txt

Then you can easily inspect the problem by searching for “1 fail” in temp.txt.

The next part is to actually fix the problem in the spec. When you believe that you have fixed the problem using your normal development workflow (changing the spec, and running to see that it passes correctly), you can invoke the shell loop again. The output file should have a different name or after deleting the previous file so you don’t get confused by previous failures. This process ensures that the problem is indeed taken care of and won’t pop up again.

Obviously if you’re doing continuous deployment, failing once every fifty times might be problematic. :)