Mutation testing is making a come back and rightfully so. Now that we have agents doing all the testing for us we need to make sure that they’ve written all necessary tests, making our test suite stronger and our project better.

Mutation testing

In a nutshell, mutation testing is a way to check if the tests you’ve written are covering all cases. The process is simple:

1. You go and change the production code by either removing code (without breaking the compilation) or altering it (for example change the conditions in an if-else statement)
2. You run the test suite. If nothing fails then your tests were not covering all cases.

Lets see an example:

	// production
	class AdmissionPolicy {
	fun canEnter(age: Int): Boolean = age >= 18
	}
	// tests
	class AdmissionPolicyTest {
	@ParameterizedTest(name = "age {0}: admission allowed = {1}")
	@CsvSource("17, false", "19, true")
	fun `admission is granded for 18 and above`(age: Int, expected: Boolean) {
	assertEquals(expected, AdmissionPolicy().canEnter(age))
	}
	}

view raw AdmissionPolicy.kt hosted with ❤ by GitHub

The admission test is making sure that the code works as expected. Anyone under 18 is being denied access and anyone over it is granted.

Lets mutate the production code by changing age >= 18 to age > 18 and run the tests again. Turns out that all tests pass despite the change. This means that if the same change is done by a dev there will be nothing stopping it to get to production. The mutation survived!

Fortunately mutation testing exposes these cases and we can patch them before they reach our users. Here the fix is adding one more test entry in the CSV source:

	class AdmissionPolicyTest {
	@ParameterizedTest(name = "age {0}: admission allowed = {1}")
	@CsvSource("17, false", "18, true", "19, true")
	fun `admission is granded for 18 and above`(age: Int, expected: Boolean) {
	assertEquals(expected, AdmissionPolicy().canEnter(age))
	}
	}

view raw AdmissionPolicyTest2.kt hosted with ❤ by GitHub

PIT

Doing that by hand is tedious so the community has written libraries that mutate the code on their own. In the JVM world the preferred library is PIT. PIT supports lots of mutations and what makes it special is that it does not mutate your code directly. It changes the produced byte code which makes the entire procedure faster.

Unfortunately when it comes to Kotlin, the library’s advantage is also its disadvantage. In order to be able to run in JVM, Kotlin’s compiler produces Java compatible bytecode which means it adds more code to the original file:

	// kotlin
	fun greet(name: String): String {
	return "Hello, $name"
	}
	// java compatible translation
	public final class ExampleKt {
	@NotNull
	public static final String greet(@NotNull String name) {
	return "Hello, " + name;
	}
	}

view raw jvm-compatibility.java hosted with ❤ by GitHub

This means that PIT might mutate code that was not there on the first place and produce a report with a lot of noise.

Clearing the report

At this point the only thing we can do is to clear the report. This means that for every entry in it we have to:

1. Check that it is one of the survived mutations
2. Make sure that the change is on a piece of code that exists on the source file

If both are true then we keep the entry and proceed with a fix. If we are dealing with a couple of mutations then this can be done manually. In any other case we have to automate things:

This is what I’ve ended up with:

Step #1: Keep survived mutations

PIT adds all cases in the same report so the first step was to create a script that will take the initial report as input and produce a second report that contains only the survived mutations.

Step #2: Clean data

To be honest this step is optional. Every entry in the report contains lots of data which makes things difficult to parse manually and reason with. This step uses a script to keep only the information we find important and put them in a single line. Easier to read and comprehend.

Step #3: Remove invalid mutations – inserted methods

Every entry has a pair of “mutatedMethod” and “sourceFile” values. This step uses a script that extracts these values and uses them to make an actual search of the method in the source file. If there are no findings then the mutation is not valid (not on our original code) and gets ignored.

An example of such a value is: < mutatedMethod >invokeSuspend</ mutatedMethod > where invokeSuspend is something that the compiler adds whenever we have coroutines in our Kotlin code.

Step #4: Prepare data for removing rest of invalid mutations

Apart from inserted methods we might have insertions of code in existing methods. This makes the cleanup a bit more difficult since we must do an actual check of both our source code and the change the library did.

For example one of the mutations is described by the tool as:

removed call to kotlin/ResultKt::throwOnFailure

In order to see if this is a valid mutation we need to make sure that the original code had a call to the throwOnFailure in the first place. If not then this is invalid.

Since this cannot be done manually we are going to use Jev and to make things easier this step prepares the state that we are going to send to it. So for the aforementioned mutation the preparation is to create a file with the following:

	Code:
	override suspend fun add(transaction: Transaction) {
	mutex.withLock {
	transactions.add(transaction)
	}
	}
	Change:
	removed call to kotlin/ResultKt::throwOnFailure

view raw jev-state.txt hosted with ❤ by GitHub

Everything is being handled by an agent that is being prompted to find the entry’s method, extract it from the rest of the code and put it in a file along side the mutation’s description.

Step #5: Jev checks mutation’s validity

This step uses a script that for every created file it makes a call to Jev and ask it to confirm if the mutation is valid. In the call it passes the file’s content as the state and asks a Noul question:

Given the Kotlin code before any change, is it possible for the mentioned change to be applied in the code?

Jev answers with a value between 0 and 1. If the value is smaller than 0.5 then the file gets ignored. Any other case moves the file to the next step.

Step #6: Kill mutations!

Here we pass our findings to an agent and prompt it to fix the mutation’s testing results:

Fix the valid surviving PIT mutations by adding or strengthening regression tests that kill them. Do not apply the mutations to production code or change production semantics…