Mutation Testing Guide: What You Should Know

If you’re just starting to learn the concept of Mutation Testing, this is the place to find out What is Mutation, What is Mutation Testing, and the following.

What is Mutation

Mutation (fault) can be defined as small but deliberate modification in a source code; these small modifications are planned to typical low-level errors which happen at the time of the coding process.

The source code gets mutated i.e minute changes are made in the program. We will see a difference once, test data is run through the mutant. Mutants are also called mutant programs.

Mutant programs can be defined as simply the mutated version of the source code. In other words, it injects faults into the program by rewriting the source code. Few faults are seeded into the source code to find the ambiguities inside the code. When the test data is run through the mutant program, we should get a different result from the source code.

Mutants State

Survived Mutants: Mutants that are still alive i.e not detected when the test is executed is called survived mutants. They are also known as live mutants. Ideally, they should have been identified through the test case.
Killed Mutants: Mutants that are identified while testing is killed mutants. This happens when we get a different output from the source code.
Equivalent Mutants: These mutants have to save meaning as the source code with a different syntax. They are still alive after the test data is run through them, they are related to living mutants.
No coverage: In this case, the mutant survived because no tests were executed for this mutant. This mutant is located in a part of the code not hit by any of your tests. This means that our test case missed to cover it. it.
Timeout The running of tests with this mutant active resulted in a timeout. For example, the mutant resulted in an infinite loop in your code. Don’t spend too much attention on this mutant. It is counted as “detected”. The logic here is that if this mutant were to be injected into your code, your CI build would detect it because the tests will never complete.
Runtime error: The running of the tests resulted in an error (rather than a failed test). This can happen when the test runner fails. For example, when a test runner throws an OutOfMemoryError or for dynamic languages where the mutant resulted in unparsable code. Don’t spend too much attention looking at this mutant. It is not represented in your mutation score.
Compile error: This state happens when it is a compiled language. The mutant resulted in a compiler error. It is not represented in the mutation score, so you don’t need to spend too much attention looking at this mutant.
Ignored: We can see this state when the user sets configurations for it to be ignored. It will be shown up in reports but it won’t affect the mutation score.

What is Mutation Testing

Mutation testing is a white box testing technique in software testing where we insert certain statements in the source code (errors) purposefully to check if the existing test case can detect those issues.

This type of testing helps both the tester and developers in developing a better test suite and to locate weaknesses in the test data.

It is a unit testing method. It uses a fault injection method to insert mutants into the program.

In simple words, mutual testing tests the efficiency of the test cases.

Mutation test can be done using four simple steps, they are as follows

Modify the source code with mutants.
Develop test cases for those specific areas.
Run the test cases for both original and mutant programs.
Compare the results

The goals of mutation testing are:

To find a set of code that is not proper.
To identify hidden defects that can’t be detected using other testing methods.
To identify new kinds of errors or bugs.
To calculate the mutation score.
To check both the coverage and effectiveness of the test cases.

Mutation Testing Types

Mutation tests are of several types, each of them has its own goal and usage.

Value mutation involves interchanging small values with big values i.e modification of different values
Statement mutation is implemented by altering statements in the source code.
Decision mutation identifies defects by changing the logical, arithmetic, and relational operators of the defects to be identified.

Value Mutation

In this type of mutation, the values of the primary parameter are changed to check the output of the program.

Original Code:

int num = 101;
int x = 98765432;
int y = 12345678;
int z = (x + y) % num;

Mutant Code:

int num = 1000000001;
int x = 98765432;
int y = 12345678;
int z = (x + y) % num;

Statement Mutation

In this type of mutation, a statement can be edited, deleted or rearranged. This modification in the code has a certain impact on the output.

Original code:

if(x < y)
z = 20;
else
z = 30;

Mutant code:

if(x < y)
w = 20;
else
w = 30;

Decision Mutation

In this type of mutation logical or arithmetic operators are changed to detect errors in the program.

Original Code:

if(x > y)
z = 20;
else
z = 30;

Mutated Code:

if(x < y)
z = 20;
else
z = 30;

Mutation Score:

If we kill all the mutants in the program, we can say that we have provided complete coverage to the program.

Mutation score can be defined as the amount of coverage.
Here we look at mutants as a test requirement.
We use random sampling to assign the number of mutants to the program hence random sampling

The mutation score can be defined as the percentage of killed mutants with the total number of mutants.

Mutation Score = (Killed Mutants / Total number of Mutants) * 100

If the mutation score is 100% then the test cases will be adequate. This an effective approach to measure the adequacy of the test cases. But generating mutants and executing test cases for each mutant program can be intensive and it requires huge cost.

Other mutation metrics

Detected

It is the number of mutants that are detected by our test i.e the killed mutants.

Killed mutants +Timeout

Undetected

It is the number of mutants that are not detected by our test, i.e the survived mutants.

Survived mutants + No Coverage

Covered

It is the number of mutants our tests produced code coverage for.

Detected mutants + Survived mutants

Valid

It is the number of valid mutants, they didn’t in a compile error or runtime error

Detected mutants + Undetected mutants

Invalid

It is the number of all invalid mutants ie. they couldn’t be tested as they produced a compile error or a runtime error.

Runtime errors + Compile errors

Total mutants

It contains all mutants, it focuses on everything.

Valid + Invalid + Ignored

Mutation score

It calculates the percentage of the total mutants that were killed.

Detected / Valid * 100

Mutation score based on covered code

It asses the total percentage of the killed mutants based on the code coverage

Detected / Covered * 100

Incorrect syntax

They are called stillborn mutants, it is introduced as a syntax error. Usually, these errors should be detected by the compiler.

Let’s look at an example of a code where we introduce a syntax error to an original code. If we get a compiler error then the mutant is detected.

Original Code:

Read numbers
if numbers < 100
Type = Double Digit()
End if

Mutant Program:

Read numbers
if numbers %% 100
Type = Double Digit()
End if

Equivalent mutants

When a mutant has the same semantic meaning as the original code it is an equivalent mutant.

Original Code:

int i=0;
while(…)
{
….;
i++;
if(i==10)
break;
}

Mutant Code:

int i=0;
while(…)
{
….;
i++;
if(i>=10)
break;
}

Trivial Mutants

Trivial mutants as the name suggest do not do anything to the programming. Any test case can kill these mutants. If there are still test cases at the end of the testing, then it’s an invalid mutant.

Example:

Read numbers
if numbers = 10
Remove assignment statement
End if

Mutation Testing Benefits

Mutation testing helps the team by bringing attention to new kinds of errors.
Mutation testing can be very powerful as it helps in identifying hidden defects which would have been impossible to identify using the traditional testing techniques.
Mutation testing makes it easier to maintain and debug the product.
Mutation testing is the most comprehensive technique to validate any product.
Mutation testing can detect all the loopholes in the source code
Mutation testing can be very useful for assessing the effectiveness of test strategies

How to perform Mutation Testing

Step 1: The team would introduce a few mutants ie. faults to the source code. A single fault is placed in the program which helps us understand the weak areas in the program and the effectiveness of our test cases.

Step 2: Usually automation scripts are run through the program. The scripts which are used for testing the original program, are run in both the original program and mutant program. Test cases/ test scripts should be efficient enough to identify the defects in the program when both tests are run in parallel, it helps us find the exact spots where the code is weak.

Step 3: Then the team compares the results of both the original program and mutant program.

Step 4: If the test results are different i.e defects are identified, then that the mutant is killed by the test case/test script. The test case/test script is efficient enough to detect the between the original and the mutant program.

Step 5: If there is no difference between the output, the mutant is still alive. The team tries to identify the weak areas in the test cases/test script to kill the mutants the next time.

Mutation Testing Tools

Mutation testing can be extremely time consuming and complicated to execute manually. It is advisable to go for automation tools to speed up the process.

They also help in reducing the cost and time of the team. These tools will help us to process mutant generation. Few mutant testing tools are listed below:

Stryker Mutator
Jumble
PIT
Insure++.
Pester
MuClipse

Advantages of Mutation Testing

It increases the trust of the code as it translates to stable and reliable systems
Because of mutation testing, the customers get the most stable and reliable system.
Mutation testing helps the team to detect all faults in the source code
Mutation testing reveals ambiguities in the source code in the primary stages.
Mutation testing provides high coverage of the source program.
The overall quality of the software product gets increased.
Loopholes in both the code and the test data can be identified through mutation testing.
The customer gets a safe and reliable product

Disadvantages of Mutation Testing

Mutation Testing needs a lot of time and resources which increases the cost of the process.
Mutation testing can be a bit complicated to perform, so it should be automated.
As changes are done in the code level, mutation testing can’t be done manually, black-box testing can’t be done for this testing.
When one mutant is introduced, the program has to be tested from end to end to see its side effects, the automation suite should have good coverage.
Many mutant programs will demand to test against the original test suite.
As a very large number of mutants are generated using random sampling and selective mutation, it will be computationally intensive.

Conclusion

If your product needs exhaustive testing and your automation suite is efficient, you can always do mutation testing. Because it is the most comprehensive technique to test any program as it tests the product at the code level. It is used to validate the efficiency of the test cases. But it can be a time-consuming and resource-intensive activity. You can go for it only if you have the proper automation tools and requirements to do the maximum code coverage

Related posts:

Mutation Testing Guide: What You Should Know

What is Mutation

Mutants State

What is Mutation Testing