Keep Calm and Kill Mutants
Written by Alexander Shestakov   
Monday, 22 August 2016
Article Index
Keep Calm and Kill Mutants
Testing, Testing
Stricter Equality

Code is changed, running specs: 

snip8

Zero failures! It surprises me every time. After staring at other live mutants, I finally realized my mistake, thinking one test case proves correctness or can protect me from regressions.

Next attempt, I tried to write a very extensive test suite, covering every business feature one by one:

9

Let's run our friend Mutant once again:

tri10

Much better this time, 91.33%, but not perfect. Let's see why:

mut11

This time, one of Mutant's complaints was about using the .fetch method instead of just [ ] array accessor.

You might think it's not a big deal, but it's worth considering further. The difference between the semantics of [ ] and fetch is in the handling of absent values: [ ] will silently return nil, while fetch will raise an error. Instead of substituting accessors in our code to a stricter version, let's think what Mutant is actually trying to tell us.

The message is: "There might be a problem with error handling in your code or the test suite does not have a case, forcing your code to suffer from NoMethodError on nil values".

That’s fine. Let's try to write one:

snip12

It predictably fails with good old NotMethodError:

snip13

What can we do about it? Let's actually try to apply the change, suggested by Mutant. The vote method now looks like this:

snip14

 

Running tests again results in a different IndexError error:

snip15

This one is actually much better to work with. The code blows up exactly where it should, on the code containing accessing error, not later. In the previous run, it failed on << operator, which in real world examples may be far away from the place containing an error.

Now we’ve learned something useful. Let's not waste time on actually fixing it properly. Instead, I'll just pretend IndexError is the desired behavior and replace all hash and array accessing methods by fetch:

snip16

Running Mutant again:

tri17

Better results: 92.95 percent. What's next?

mut18

Surprising again. I'm convinced I can fully rely on Mutant now and will not try to double-check it once again. It looks like we did not check the case where somebody attempts to work with finished workflow, which forces @current_step variable to increase and @current_step == @steps_config.size invariant does not work any more.  

Introducing the new test and fixing code:

snip19

Mutant reports 94.17 percent of mutation coverage now, complaining about equality semantics:

mut20



Last Updated ( Wednesday, 31 August 2016 )