FlakyTest

record in io.mutastic.api

record FlakyTest(
    String test,
    List<String> failure,
    List<String> reaches,
    List<String> reachesInTest)

What a flaky test’s failure and reach say about why it’s flaky: the difference between a test that’s its own doing (an unseeded Random, the clock), which can be made to give the same answer every time, and one that reaches outside the code (the network, a database), where changing the test or the code to get past it would hide what’s wrong.

Components

test	the test, as com.example.QuoteClientTest#aPriceComesFromTheFeed
failure	the exception it failed with and its causes, outermost first, by class name; empty when the worker didn’t say
reaches	calls the code the test ran makes that give a different answer from one run to the next: new Random(), Instant.now, System.nanoTime
reachesInTest	the same, in the test’s own class: its set-up, a stand-in it starts. A test whose own code varies makes its own flakiness, even when it fails on the network: a stand-in feed with a random delay times out by the test’s doing, not the network’s.

Methods

failedOutside

boolean failedOutside()

Whether it failed with something from outside the code: an exception, or a cause of one, from the network, a database, a channel or a file, or a timeout waiting on one.

failedWith

Optional<String> failedWith()

The exception it failed with, by its simple name, if the worker said.

outsideCause

Optional<String> outsideCause()

The innermost cause from outside the code, by simple name: HttpTimeoutException.