Blog
Development

PIT Mutation Testing

-

Real world mutation testing

It’s been hard keeping up with everything else going around the world, even after eliminating the long travelling hours. But still, I do have something interesting to share from work. So far, the projects I have been assigned to, had an average testing coverage, ranging from 70-90ish%. But here at our client, we have

  • Junit tests
  • Regression tests
  • Pitests

Everything should be at 100% before it is merged into develop branch.

For us at Luminis, Java, Junit test sit in the core, so I won’t be using my word limit for that.
Regression tests is a synonym for Cucumber tests at our client. Most of us are familiar with Cucumber tests too. It allows automation of functional validation in easily readable and understandable format (like plain English) to Business Analysts, Developers, Testers, etc.
Cucumber (created in 2008) executes tests specified written in language called Gherkin. It is a plaint-text natural language (for example, English or one of other 60+ languages supported by Cucumber) with a given structure.

In a Gherkin file, non-blank lines can start with a keyword, followed by text in natural language. The main keywords are the following:

  • Feature: High-level description of the software feature to be tested. It can be seen as a use case description.
  • Scenario: Concrete example that illustrates a business rule. Scenarios follow the same pattern:

Example of a feature file:
Scenario Outline: Successful login to Searchlight as a legit Searchlight user 
    Given I am on the "Login" page 
    When I fill in "username" field with "foo@mydomain.com" 
    And I fill in "password" field with "CucumberIsMagic" 
    And I click "submit" button 
    Then I should be logged in successfully as "foo@mydomain.com"

Pitests – Now this is the interesting part. At least for me. I never came across mutation testing, i.e., a software testing in which certain statements of the source code are changed/mutated to check if the test cases can find errors in source code. The goal of Mutation Testing is ensuring the quality of test cases in terms of robustness that it should fail the mutated source code.
Traditional test coverage (i.e line, statement, branch, etc.) measures only which code is executed by your tests. It does not check that your tests are actually able to detect faults in the executed code. It is therefore only able to identify code that is definitely not tested. The most extreme examples of the problem are tests with no assertions. Fortunately these are uncommon in most code bases. Much more common is code that is only partially tested by its suite.

Pitest at work
Our code:
public boolean isPositive(int number) {
    boolean result = false;
    if (number >= 0) {
        result = true;
    }
    return result;
}

Pitest makes the following mutations of our code to test if they really do meaningful work
#1 Mutation – Changed conditional boundary (mutator)
    public boolean isPositive(int number) {
        boolean result = false;
        // mutator - changed conditional boundary
        if (number > 0) {
            result = true;
        }
        return result;    
    }

#2 Mutation – Negated conditional (mutator)
public boolean isPositive(int number) {
    boolean result = false;
    // mutator - negated conditional
    if (false) {
        result = true;
    }
    return result;
}

A Good unit test should fail (kill) all the mutations #1,#2,#3.
@Test
public void testPositive() {
    CalculatorService obj = new CalculatorService();
    assertEquals(true, obj.isPositive(10));
}

The above unit test will kill the mutation #2 (unit test is failed), but the mutation #1 is survived (unit test is passed).

Review the mutation #1 again. To fail (kill) this test (mutation), we should test the conditional boundary, the number zero.
    public boolean isPositive(int number) {
        boolean result = false;
        // mutator - changed conditional boundary
        if (number > 0) {
            result = true;
        }
        return result;       
    }
Improving the unit test by testing the number zero.

    @Test
    public void testPositive() {
        CalculatorService obj = new CalculatorService();
        assertEquals(true, obj.isPositive(10));
        //kill mutation #1
        assertEquals(true, obj.isPositive(0));
    }
Done, 100% mutation coverage now.

There are other mutation testing systems for Java, but they are not widely used. They are mostly slow, difficult to use and written to meet the needs of academic research rather than real development teams.

PIT is different. It’s fast – can analyse in minutes what would take earlier systems days; and is actively developed & supported.
The reports produced by PIT are in an easy to read format combining line coverage and mutation coverage information.


And it is quite easy to integrate with Ant, Maven, Gradle and others. There are eclipse and IntelliJ idea plugins available for use too. So, there is something new I learned at work.