Flaky tests

A flaky test passes or fails at random, with nothing changed. In mutation testing that matters more than usual: a mutant is caught when a test fails, so a test that fails by chance “catches” mutants it never checked. The score goes up for nothing, and moves from run to run with no change to the code.

mutastic looks for flaky tests in three ways. Two cost nothing and happen on their own; the third is a choice. Whatever it finds, it names the test, so you know what to fix.

Check	When	Costs
Before any mutant	Every run	Only a failing test’s reruns
Between runs	Each full run after the first, with the same code	Nothing
Confirming catches	With --confirm-catches	One more test run per catch

What you’ll see

The flaky sample in mutastic’s repository has two flaky tests on purpose. On its second run, with --confirm-catches, the end of the output has every warning:

 ! flaky: StockTest#aSaleReducesStock failed, then passed when run again; left out of this run
 ! likely flaky: StockTest#restockingWorks; with the code unchanged, whether it catches m4b6a65ac changed since the last run
 ! likely flaky: StockTest#restockingWorks; its catches of m4b6a65ac, m7610241c didn't repeat when run again

   Classes  0 clean | 1 with misses (1)
   Mutants  ✓ 4 caught · ✗ 2 missed · ○ 2 not covered (8)
  Worst at  Stock.restock() 2 ✗
     Flaky  1 test left out
 Test runs  6 (12 without coverage selection)
  Start at  17:38:28
  Duration  824ms (setup 452ms, coverage 222ms, mutants 148ms)
     Tests  2 on JDK 17 · 1 worker
   Reports  build/mutastic/mutation.json · build/mutastic/mutation.html
     Score  50.0%  ▼ 12.5 vs last run (same code)  threshold 80  · 66.7% where tests reach

  FAIL  Score below 80: 2 missed, 2 not covered.
 Next: mutastic verify <id> samples/flaky · mutastic run --only=<id> samples/flaky · mutastic explain <id> samples/flaky

• ! flaky: names a test that failed before any mutant, then passed. It’s left out of the run, and the Flaky row counts it.
• ! likely flaky: names a test whose catches can’t be trusted: once because a mutant’s result changed since the last run with the code unchanged, once because its catches didn’t repeat.
• The score fell by 14.3 points with the code unchanged (same code). That drop is the flaky test’s luck running out, not a change in the tests.

Before any mutant

mutastic runs all your tests once before it changes anything, to see which tests reach each line. They have to pass then: a mutant is only caught when a test that passed fails.

• A test that fails is run again on its own, up to three times. If it passes, it’s flaky: the run goes on without it, and names it (! flaky: …). Its catches would be luck, so none count.
• A test that fails every time stops the run with exit status 3: the tests fail without any mutant, and nothing can be judged against them. See Troubleshooting.
• A test class that fails as a whole, in a @BeforeAll say, can’t be run again on its own, so it stops the run too.

Why it’s flaky, and what to do

What to do about a flaky test depends on why it’s flaky, so mutastic says what it saw: the exception the test failed with, and anything the test, or the code it ran, reaches that answers differently every run. A seeded new Random(42) answers the same each time, so only the unseeded sources count: new Random(), ThreadLocalRandom, SecureRandom, Math.random, UUID.randomUUID, Collections.shuffle, the system clock and the now()s.

What mutastic saw	Why it’s likely flaky	What it says to do
The test’s own code reaches one, as a stand-in service with a random delay does	Its own doing, even when it fails on the network	Make it the same every run, and if it stands in for something real, like a slow service, keep that checked by a test of its own
It failed with an exception from the network, a database or a file, and nothing in the test varies	The environment: a shared service, a database that’s sometimes slow	Leave the test and the code alone, since changing them to get past it would hide it; tell the person, and run without it with --exclude-tests
The code it ran reaches one	The code’s own randomness or clock	Make it the same every run, through the code’s own seam (a Random or a Clock passed in), so long as the test still checks what it did
None of these	Unknown	Leave it unless the cause is plain in the code, and say it’s flaky

Whichever it is, never remove, disable or loosen the test to get a run through. The agent format gives the reason and the advice beside the test’s name, the human formats the reason (; it reaches new Random()), and the JSON Lines run-finished event has the evidence for each, as flakyEvidence.

The advice came from watching coding agents work on flaky tests. Told only to fix a flaky test, they did the right thing when the flakiness was the test’s own, a stand-in service with a random delay, and made it deterministic. Against a shared service that was sometimes too slow, they wrapped every call in a retry, which turns the suite green and hides that the client times out. Told instead that the failure came from outside and nothing in the test varies, they left the test alone and asked the person what to do.

Leaving a flaky test out has a price you’ll see: lines that only it reached are now reached by nothing. Their mutants are counted as not covered, under the heading “no test reaches these lines, or only a flaky one does”, and they lower the score until the test is fixed.

Between runs

Each full run keeps its results in last-run.tsv, in the module’s work directory beside the reports (build/mutastic/, or target/mutastic/ for Maven), and the next run compares with it. The score line says what changed in between: same code, code changed or mutastic changed.

With the same code, tests and mutastic, every mutant should be caught or missed exactly as before. One that changed result points at the test that caught it, in whichever run it was caught: that test is named as ! likely flaky: …, with the mutants whose results it changed. With anything changed, a different result is just the change at work, and nothing is named.

Only a full run compares and records: a run of one mutant (--only, verify, explain) says nothing about the others, and a run of a change (--since) has a different score by design.

Confirming catches

With --confirm-catches (or confirmCatches in a build plugin), each test that catches a mutant is run once more against it. A catch that repeats counts. One that doesn’t came from a flaky test: it doesn’t count, and the mutant’s other covering tests decide whether it’s caught. The test is named as ! likely flaky: …, and the mutant’s card says so beside its covering tests (… failed once, then passed: flaky).

It costs one more test run for every catch, so the more mutants your tests catch, the longer a run takes. It’s worth it in CI, where a score has to hold, or once a run has named a likely flaky test.

A test that fails the same way everywhere

One more pattern earns a note: rather than a warning: a test that catches mutants in several unrelated methods, each time with the same exception thrown in the test’s own code. It may be flaky, failing whatever the change. Or it may just call a lot of code. Run it on its own a few times to tell which.

For agents and scripts

The same evidence is in the other formats:

• --format=agent writes each as a line of its own (flaky: …, likely flaky: …), with what to do next.
• --format=jsonl’s run-finished event lists each likely flaky test, its mutants and its reasons (changed-since-last-run, catch-not-repeated). A mutant whose catch didn’t repeat says which test made it, as unconfirmedCatchBy.

Fixing a flaky test

mutastic tells you which test; why it’s flaky is in the test or the code it runs. The usual causes:

• Order. It depends on another test running first, or not running: shared static state, a system property, a database row. Run it on its own, and in a different order.
• Time. It reads the clock, or sleeps and hopes a background thread has finished. Pass a clock in, and wait for the thread rather than for time.
• Randomness. It uses random input without a fixed seed.
• The outside world. A network call, a port, a file left behind by an earlier run.

Until it’s fixed, a test class that can’t run reliably under mutastic, with a race or a fixed port, can be left out of every run: --exclude-tests=CheckoutRaceTest, by its name or full name. The summary names what was left out (Left out CheckoutRaceTest (--exclude-tests)), and the lines only it reached count as not covered, as for a flaky test mutastic leaves out itself.

Once it’s fixed, run mutastic again. A test that was left out is back in, and the mutants it covers are judged by it again. A test named likely flaky stops being named once its catches repeat.