pitest | State of the art mutation testing system for the JVM | Testing library

Pitest does not currently support static mocking with mockito. I'll see if it could be supported, but it likely to be a complex task. Support for Powermock required dark magics (rewriting the bytecode of the bytcode manipulation library it uses), and was always brittle and easily broken by new Powermock releases.

A better long term solution would be to remove the need for static mocking from the test suite. Although it does have some use cases, it is most often a red flag for design issues.

Pitest mutates bytecode rather than source code. The mutators attempt to describe the equivalent change in the source file, but sometimes this is less than straightforward.

These mutations look to have been generated by the 'rv' mutators. These are generally of lower quality than the standard set and are not reccomended for general use. Do not use them unless you have a particular reason to do so.

The "incremented a local variable" should really read "incremented a the local variable or paramter". The operator would need to do additional analysis to work out which.

The not equal to greater than mutation will have mutated a IFNE instruction in the bytecode. In the simplest case this will map to a != check as the mutation description suggests, but the compiler may choose to use this instruction when generated other logic constructs (in this case equality checks combined with an &&). The description is certainly misleading, updating the mutator to make it more accurate would however involve writing the large part of a decompiler.

Turns out our remote environment was running the tests a slightly different way and was using a plugin for pitest that was not explicitly defined in our POM. The plugin was Alex Victoor's pitest-cucumber-plugin which includes a compatibility matrix that I was able to use to resolve the issue.

As pointed by henry's comment, I fixed the issue by upgrading to the latest version of pitest. These versions worked for me:

According to the Maven plugin quickstart documentation for Pitest there's a excludedTestClasses configuration parameter where you can specify which tests should be excluded. So assuming all your integration tests are in the same package you should be able to do something like this:

Various console apps either default to showing with Ansi terminal codes, or without. The Symfony console command has a --no-ansi option to turn it off.

An issue at https://github.com/symfony/cli/issues/222 also suggested the env var NO_COLOR, which does also work.

how to actually setup mutation testing in maven

basically... follow this tutorial: https://pitest.org/quickstart/maven/

skip the installation part in the tutorial

download apache-maven-3.8.1 [or whatever is the latest version]

add the downloaded file to your path via command line:

The call to scala/MatchError is synthesized by the compiler for the case where the match fails (it's the call to MatchError's constructor which gets thrown if none of the cases in the pattern match apply). Since your pattern match is in fact exhaustive, there's no way for the match to fail, ergo there's no way in Scala to force the match to fail (this is assuming that you're using the standard library List; if you're using your own implementation of List, perhaps it's not sealed?)

It's perhaps a Scala compiler bug that it generates bytecode for failed pattern matches which can't fail.

I'm not familiar with pitest, but it's likely that it doesn't understand Scala (or the higher level semantic meaning of the bytecode emitted by the Scala compiler); tools which rely on bytecode manipulation are likely to not always work as they would for Java with Scala.

After a hell of a lot of debugging and trying things out, I found out that test methods were not found because the junit dependency hamcrest-core was missing.

For everyone else experiencing a problem like this, try debugging the following method: