Automatic Testing - Programmers Are Still The Problem
Automatic Testing - Programmers Are Still The Problem
Written by Alex Armstrong   
Friday, 16 June 2017

White box test generator tools can help you find bugs automatically by generating combinations of inputs that are likely to give the wrong answer. However there is a problem - how do you know when you have the wrong answer?

A recent study by David Hon€ and Zoltan Micskei of Budapest University of Technology and Economics puts automatic testing under the spotlight.

White-box test generator tools rely only on the code under test to select test inputs, and capture the implementation’s output as assertions. If there is a fault in the implementation, it could get encoded in the generated tests. Tool evaluations usually measure fault-detection capability using the number of such fault-encoding tests. However, these faults are only detected, if the developer can recognize that the encoded behavior is faulty.

They carried out a study using 54 grad students and two open source projects and the IntelliTest Tool which is still better known as Pex. The tool generated test cases and the subjects were asked to identify when the output was indicative of a fault. The results were some what surprising.

ŒThe results showed that participants incorrectly classi€ed a large number of both fault-encoding and correct tests (with median misclassi€cation rate 33% and 25% respectively). Œus the real fault-detection capability of test generators could be much lower than typically reported, and we suggest to take this human factor into account when evaluating generated white-box tests.

You can see the confusion matrix for the NBitcoin project's tests:


FP=False Positive, FN=False Negative, TP=True Positive, TN=True Negative

What this means is that in practice white box testing is subject to programmer errors in its own right. By examining videos of the subjects trying to interpret the tests the researchers note that they tended to use debugging methods to further explore the code and clarify the meaning of the test. An "exit" survey also indicates the sorts of difficulties the subjects felt they had with the task:

  • “Deciding whether a test is OK or wrong when it tests an unspecifi€ed case, e.g. comparing with null, or equality of null.”

  • “Distinguishing between the variables was difficult, e.g between assetMoney, assetMoney1, assetMoney2.”

  • “Tests should compare less with null and objects with themselves.”

  • “I think that some assertions are useless, and not asserting ’real problems’, just some technical details.”

  • “Generated test cases doesn’t separated into Arrange, Act, Assert and should create more private methods for these concerns.”

  • “Generate comments into tests describing what happening.”

The key finding is:

ŒThe implication of the results is that the actual fault-€nding capabilities of the test generator tools could be much lower than reported in technology-focused experiments. 

Of course it would be better if the programmer could be taken out of the loop, but this would require the white box tester to work out what the result of the test should be and this would require a lot of AI.

More Information

Classifying the Correctness of Generated White-Box Tests: An Exploratory Study

Related Articles

Code Digger Finds The Values That Break Your Code

Code Hunt - New Coding Game From Microsoft Research

Debugging and the Experimental Method       

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on, Twitter, FacebookGoogle+ or Linkedin.



Fear And Loathing In The Cloud

The cloud has many advantages, but there is one very big downside - you hand over control of your major assets to a third party that you simply have to trust. Is the thought that you could be held to  [ ... ]

Neural Networks Learn To Remove Noise Without Seeing A Good Image

Don't you just hate film grain? Except, of course, when you don't and set it as a special effect on your digital camera. NVIDIA now has a tool that will remove grain, noise and all manner of defects,  [ ... ]

More News





or email your comment to:

Last Updated ( Friday, 16 June 2017 )

RSS feed of news items only
I Programmer News
Copyright © 2018 All Rights Reserved.
Joomla! is Free Software released under the GNU/GPL License.