Keep Calm and Kill Mutants
Written by Alexander Shestakov   
Monday, 22 August 2016
Article Index
Keep Calm and Kill Mutants
Testing, Testing
Stricter Equality

Code is changed, running specs: 


Zero failures! It surprises me every time. After staring at other live mutants, I finally realized my mistake, thinking one test case proves correctness or can protect me from regressions.

Next attempt, I tried to write a very extensive test suite, covering every business feature one by one:


Let's run our friend Mutant once again:


Much better this time, 91.33%, but not perfect. Let's see why:


This time, one of Mutant's complaints was about using the .fetch method instead of just [ ] array accessor.

You might think it's not a big deal, but it's worth considering further. The difference between the semantics of [ ] and fetch is in the handling of absent values: [ ] will silently return nil, while fetch will raise an error. Instead of substituting accessors in our code to a stricter version, let's think what Mutant is actually trying to tell us.

The message is: "There might be a problem with error handling in your code or the test suite does not have a case, forcing your code to suffer from NoMethodError on nil values".

That’s fine. Let's try to write one:


It predictably fails with good old NotMethodError:


What can we do about it? Let's actually try to apply the change, suggested by Mutant. The vote method now looks like this:



Running tests again results in a different IndexError error:


This one is actually much better to work with. The code blows up exactly where it should, on the code containing accessing error, not later. In the previous run, it failed on << operator, which in real world examples may be far away from the place containing an error.

Now we’ve learned something useful. Let's not waste time on actually fixing it properly. Instead, I'll just pretend IndexError is the desired behavior and replace all hash and array accessing methods by fetch:


Running Mutant again:


Better results: 92.95 percent. What's next?


Surprising again. I'm convinced I can fully rely on Mutant now and will not try to double-check it once again. It looks like we did not check the case where somebody attempts to work with finished workflow, which forces @current_step variable to increase and @current_step == @steps_config.size invariant does not work any more.  

Introducing the new test and fixing code:


Mutant reports 94.17 percent of mutation coverage now, complaining about equality semantics:


Last Updated ( Wednesday, 31 August 2016 )