Java edition

Schemas and stability

A CI step that fails the build, a dashboard, a coding agent’s plugin: each reads what mutastic found, and each breaks if what it reads moves under it. So what mutastic writes for another tool comes with a JSON Schema and a promise. Within a version, a field keeps its name and its type, and anything new is optional, so a reader that takes what it knows and lets the rest pass keeps working, release after release.

Events

--format=jsonl writes one JSON object per line, as the run goes, so a reader can act on each mutant as it finishes. This is the Quickstart’s run of the pricing sample, as mutastic run samples/pricing --format=jsonl writes it:

{"schema":"mutastic.events/v1","edition":"java","reason":"run-started","project":"pricing","rule":"add or change tests only; never change production code to make a mutant go away: tell the person","decline":"if a mutant's case can't happen in practice, or the change is one no caller can see, don't contrive a test: propose ignoring it, for a person to accept, as the kind it is (equivalent, unreachable, telemetry, deprecated or accepted-risk), with the reason in two sentences: the one case where it differs, and why that can't happen"}
{"schema":"mutastic.events/v1","edition":"java","reason":"mutant-finished","id":"m855c8e3c","fingerprint":"855c8e3ca8a39847","status":"Survived","operator":"mutastic/boundary","docs":"https://docs.mutastic.io/concepts/operators/#boundary","location":{"file":"src/main/java/com/example/pricing/Pricing.java","line":17,"class":"com.example.pricing.Pricing","method":"discount(I)I"},"mutation":{"original":">=","replacement":">","description":"`>=` becomes `>`","applied":{"level":"bytecode","text":"IF_ICMPLT -> IF_ICMPLE"}},"coveredBy":["com.example.pricing.PricingTest#bigOrdersGetTenPercentOff","com.example.pricing.PricingTest#smallOrdersPayFullPrice"],"inputs":{"total":[50,250]},"suggestedTest":{"testClass":"PricingTest","name":"discountAtTheTotalEdge","code":"@Test\nvoid discountAtTheTotalEdge() {\n  assertEquals(10, new Pricing().discount(100)); // today 10 · with this mutant 0\n}","call":"new Pricing().discount(100)","today":"10","withMutant":"0"},"suggestion":{"call":"new Pricing().discount(100)","parameters":["total"],"arguments":[100],"receiver":[{"type":"Pricing","arguments":[],"parameters":[]}],"today":"10","withMutant":"0"},"testsRun":2,"durationMs":16,"doneWhen":"mutastic verify m855c8e3c samples/pricing passes","reproduce":"mutastic run --only=m855c8e3c samples/pricing","ignore":"mutastic ignore m855c8e3c --reason=\"<why>\" samples/pricing","propose":"mutastic ignore m855c8e3c --propose --as=<equivalent|unreachable|…> --reason=\"Differs only when <case>. That can't happen because <why>.\" samples/pricing","rendered":"missed m855c8e3c: no test checks `discount()` at `total == 100`\n   at src/main/java/com/example/pricing/Pricing.java:17\n   - if (total >= 100) {\n   + if (total > 100) {\n   covered by: PricingTest#bigOrdersGetTenPercentOff, PricingTest#smallOrdersPayFullPrice; all still pass with the change\n   evidence: every test calls `discount()` away from the boundary\n   evidence: inputs seen: 50, 250 (none is 100)\n   fix: test `discount()` with `total = 100` and assert the result\n   test to add in PricingTest:\n     @Test\n     void discountAtTheTotalEdge() {\n       assertEquals(10, new Pricing().discount(100)); // today 10 · with this mutant 0\n     }\n     // check today's value is the one you intend: the test pins what the code does now\n   done when: mutastic verify m855c8e3c samples/pricing passes\n   reproduce: mutastic run --only=m855c8e3c samples/pricing\n   or propose: mutastic ignore m855c8e3c --propose --as=<equivalent|unreachable|…> --reason=\"Differs only when <case>. That can't happen because <why>.\" samples/pricing\n"}
{"schema":"mutastic.events/v1","edition":"java","reason":"mutant-finished","id":"m53e1dc80","fingerprint":"53e1dc806373872b","status":"Killed","operator":"mutastic/remove-conditional","docs":"https://docs.mutastic.io/concepts/operators/#remove-conditional","location":{"file":"src/main/java/com/example/pricing/Pricing.java","line":17,"class":"com.example.pricing.Pricing","method":"discount(I)I"},"mutation":{"original":">=","description":"`total >= 100` is always false","applied":{"level":"bytecode","text":"IF_ICMPLT -> always jump"}},"coveredBy":["com.example.pricing.PricingTest#bigOrdersGetTenPercentOff","com.example.pricing.PricingTest#smallOrdersPayFullPrice"],"caughtBy":"com.example.pricing.PricingTest#bigOrdersGetTenPercentOff","caughtWith":"org.opentest4j.AssertionFailedError","failureOrigin":"assertion","inputs":{"total":[50,250]},"testsRun":1,"durationMs":16}
{"schema":"mutastic.events/v1","edition":"java","reason":"mutant-finished","id":"mfb551430","fingerprint":"fb5514301b9d532a","status":"Killed","operator":"mutastic/remove-conditional","docs":"https://docs.mutastic.io/concepts/operators/#remove-conditional","location":{"file":"src/main/java/com/example/pricing/Pricing.java","line":17,"class":"com.example.pricing.Pricing","method":"discount(I)I"},"mutation":{"original":">=","description":"`total >= 100` is always true","applied":{"level":"bytecode","text":"IF_ICMPLT -> never jump"}},"coveredBy":["com.example.pricing.PricingTest#bigOrdersGetTenPercentOff","com.example.pricing.PricingTest#smallOrdersPayFullPrice"],"caughtBy":"com.example.pricing.PricingTest#smallOrdersPayFullPrice","caughtWith":"org.opentest4j.AssertionFailedError","failureOrigin":"assertion","inputs":{"total":[50,250]},"testsRun":2,"durationMs":14}
{"schema":"mutastic.events/v1","edition":"java","reason":"mutant-finished","id":"mabd5e44c","fingerprint":"abd5e44c174dc289","status":"Killed","operator":"mutastic/math","docs":"https://docs.mutastic.io/concepts/operators/#math","location":{"file":"src/main/java/com/example/pricing/Pricing.java","line":18,"class":"com.example.pricing.Pricing","method":"discount(I)I"},"mutation":{"original":"/","replacement":"*","description":"`/` becomes `*`","applied":{"level":"bytecode","text":"IDIV -> IMUL"}},"coveredBy":["com.example.pricing.PricingTest#bigOrdersGetTenPercentOff"],"caughtBy":"com.example.pricing.PricingTest#bigOrdersGetTenPercentOff","caughtWith":"org.opentest4j.AssertionFailedError","failureOrigin":"assertion","inputs":{"total":[250]},"testsRun":1,"durationMs":7}
{"schema":"mutastic.events/v1","edition":"java","reason":"mutant-finished","id":"mf7bb3882","fingerprint":"f7bb3882f91a30ff","status":"Killed","operator":"mutastic/return-value","docs":"https://docs.mutastic.io/concepts/operators/#return-value","location":{"file":"src/main/java/com/example/pricing/Pricing.java","line":18,"class":"com.example.pricing.Pricing","method":"discount(I)I"},"mutation":{"replacement":"return 0;","description":"returns `0` instead","applied":{"level":"bytecode","text":"IRETURN -> return 0"}},"coveredBy":["com.example.pricing.PricingTest#bigOrdersGetTenPercentOff"],"caughtBy":"com.example.pricing.PricingTest#bigOrdersGetTenPercentOff","caughtWith":"org.opentest4j.AssertionFailedError","failureOrigin":"assertion","inputs":{"total":[250]},"testsRun":1,"durationMs":6}
{"schema":"mutastic.events/v1","edition":"java","reason":"mutant-finished","id":"m3264f639","fingerprint":"3264f6398de7f1d8","status":"Killed","operator":"mutastic/return-value","docs":"https://docs.mutastic.io/concepts/operators/#return-value","location":{"file":"src/main/java/com/example/pricing/Pricing.java","line":20,"class":"com.example.pricing.Pricing","method":"discount(I)I"},"mutation":{"replacement":"return 1;","description":"returns `1` instead","applied":{"level":"bytecode","text":"IRETURN -> return 1"}},"coveredBy":["com.example.pricing.PricingTest#smallOrdersPayFullPrice"],"caughtBy":"com.example.pricing.PricingTest#smallOrdersPayFullPrice","caughtWith":"org.opentest4j.AssertionFailedError","failureOrigin":"assertion","inputs":{"total":[50]},"testsRun":1,"durationMs":7}
{"schema":"mutastic.events/v1","edition":"java","reason":"mutant-finished","id":"m2de3396d","fingerprint":"2de3396dd2bf9472","status":"Killed","operator":"mutastic/boundary","docs":"https://docs.mutastic.io/concepts/operators/#boundary","location":{"file":"src/main/java/com/example/pricing/Pricing.java","line":25,"class":"com.example.pricing.Pricing","method":"shipping(I)I"},"mutation":{"original":"<","replacement":"<=","description":"`<` becomes `<=`","applied":{"level":"bytecode","text":"IF_ICMPGE -> IF_ICMPGT"}},"coveredBy":["com.example.pricing.PricingTest#shippingIsFreeFromTheThreshold"],"caughtBy":"com.example.pricing.PricingTest#shippingIsFreeFromTheThreshold","caughtWith":"org.opentest4j.AssertionFailedError","failureOrigin":"assertion","inputs":{"total":[39,40]},"testsRun":1,"durationMs":7}
{"schema":"mutastic.events/v1","edition":"java","reason":"mutant-finished","id":"m73f3386e","fingerprint":"73f3386e9057735e","status":"Killed","operator":"mutastic/remove-conditional","docs":"https://docs.mutastic.io/concepts/operators/#remove-conditional","location":{"file":"src/main/java/com/example/pricing/Pricing.java","line":25,"class":"com.example.pricing.Pricing","method":"shipping(I)I"},"mutation":{"original":"<","description":"`total < freeShippingFrom` is always false","applied":{"level":"bytecode","text":"IF_ICMPGE -> always jump"}},"coveredBy":["com.example.pricing.PricingTest#shippingIsFreeFromTheThreshold"],"caughtBy":"com.example.pricing.PricingTest#shippingIsFreeFromTheThreshold","caughtWith":"org.opentest4j.AssertionFailedError","failureOrigin":"assertion","inputs":{"total":[39,40]},"testsRun":1,"durationMs":6}
{"schema":"mutastic.events/v1","edition":"java","reason":"mutant-finished","id":"m091729b1","fingerprint":"091729b183cefc89","status":"Killed","operator":"mutastic/remove-conditional","docs":"https://docs.mutastic.io/concepts/operators/#remove-conditional","location":{"file":"src/main/java/com/example/pricing/Pricing.java","line":25,"class":"com.example.pricing.Pricing","method":"shipping(I)I"},"mutation":{"original":"<","description":"`total < freeShippingFrom` is always true","applied":{"level":"bytecode","text":"IF_ICMPGE -> never jump"}},"coveredBy":["com.example.pricing.PricingTest#shippingIsFreeFromTheThreshold"],"caughtBy":"com.example.pricing.PricingTest#shippingIsFreeFromTheThreshold","caughtWith":"org.opentest4j.AssertionFailedError","failureOrigin":"assertion","inputs":{"total":[39,40]},"testsRun":1,"durationMs":6}
{"schema":"mutastic.events/v1","edition":"java","reason":"mutant-finished","id":"m39e3e756","fingerprint":"39e3e756ddc40f5f","status":"Killed","operator":"mutastic/return-value","docs":"https://docs.mutastic.io/concepts/operators/#return-value","location":{"file":"src/main/java/com/example/pricing/Pricing.java","line":26,"class":"com.example.pricing.Pricing","method":"shipping(I)I"},"mutation":{"replacement":"return 0;","description":"returns `0` instead","applied":{"level":"bytecode","text":"IRETURN -> return 0"}},"coveredBy":["com.example.pricing.PricingTest#shippingIsFreeFromTheThreshold"],"caughtBy":"com.example.pricing.PricingTest#shippingIsFreeFromTheThreshold","caughtWith":"org.opentest4j.AssertionFailedError","failureOrigin":"assertion","inputs":{"total":[39,40]},"testsRun":1,"durationMs":7}
{"schema":"mutastic.events/v1","edition":"java","reason":"mutant-finished","id":"m3b5420b6","fingerprint":"3b5420b6979d0ee0","status":"Killed","operator":"mutastic/return-value","docs":"https://docs.mutastic.io/concepts/operators/#return-value","location":{"file":"src/main/java/com/example/pricing/Pricing.java","line":28,"class":"com.example.pricing.Pricing","method":"shipping(I)I"},"mutation":{"replacement":"return 1;","description":"returns `1` instead","applied":{"level":"bytecode","text":"IRETURN -> return 1"}},"coveredBy":["com.example.pricing.PricingTest#shippingIsFreeFromTheThreshold"],"caughtBy":"com.example.pricing.PricingTest#shippingIsFreeFromTheThreshold","caughtWith":"org.opentest4j.AssertionFailedError","failureOrigin":"assertion","inputs":{"total":[39,40]},"testsRun":1,"durationMs":7}
{"schema":"mutastic.events/v1","edition":"java","reason":"run-finished","jdk":17,"toolchain":{"name":"jdk","version":"17"},"mutants":11,"killed":10,"survived":1,"timeout":0,"noCoverage":0,"ignored":0,"score":90.9,"threshold":80,"passed":true,"coveredScore":90.9,"tests":3,"workers":1,"workersStarted":1,"coverageMs":300,"flaky":[],"flakyEvidence":[],"likelyFlaky":[],"unrelatedCatches":[],"reports":["build/mutastic/mutation.json","build/mutastic/mutation.html"],"staleIgnores":[],"durationMs":1364}

Each line says which format it is, and its major version, in schema: mutastic.events/v1. Its reason says which event it is:

`reason`	When
`run-started`	First, once the build has been read
`mutant-finished`	For each mutant, as it finishes
`mutant-revised`	Once every mutant has its result, for each mutant a test reached that then failed on its own with no mutant in: its result again, without that test, which replaces the first
`run-finished`	Last: the totals, the score and the verdict
`mutant-explained`	Only from `mutastic explain ID --format=jsonl`, after the other three for its one mutant: what the text’s About this mutant says, a field for each row (One mutant, explained)

A missed mutant

The one mutant the tests missed, laid out:

{
  "schema" : "mutastic.events/v1",
  "edition" : "java",
  "reason" : "mutant-finished",
  "id" : "m855c8e3c",
  "fingerprint" : "855c8e3ca8a39847",
  "status" : "Survived",
  "operator" : "mutastic/boundary",
  "docs" : "https://docs.mutastic.io/concepts/operators/#boundary",
  "location" : {
    "file" : "src/main/java/com/example/pricing/Pricing.java",
    "line" : 17,
    "class" : "com.example.pricing.Pricing",
    "method" : "discount(I)I"
  },
  "mutation" : {
    "original" : ">=",
    "replacement" : ">",
    "description" : "`>=` becomes `>`",
    "applied" : {
      "level" : "bytecode",
      "text" : "IF_ICMPLT -> IF_ICMPLE"
    }
  },
  "coveredBy" : [ "com.example.pricing.PricingTest#bigOrdersGetTenPercentOff", "com.example.pricing.PricingTest#smallOrdersPayFullPrice" ],
  "inputs" : {
    "total" : [ 50, 250 ]
  },
  "suggestedTest" : {
    "testClass" : "PricingTest",
    "name" : "discountAtTheTotalEdge",
    "code" : "@Test\nvoid discountAtTheTotalEdge() {\n  assertEquals(10, new Pricing().discount(100)); // today 10 · with this mutant 0\n}",
    "call" : "new Pricing().discount(100)",
    "today" : "10",
    "withMutant" : "0"
  },
  "suggestion" : {
    "call" : "new Pricing().discount(100)",
    "parameters" : [ "total" ],
    "arguments" : [ 100 ],
    "receiver" : [ {
      "type" : "Pricing",
      "arguments" : [ ],
      "parameters" : [ ]
    } ],
    "today" : "10",
    "withMutant" : "0"
  },
  "testsRun" : 2,
  "durationMs" : 16,
  "doneWhen" : "mutastic verify m855c8e3c samples/pricing passes",
  "reproduce" : "mutastic run --only=m855c8e3c samples/pricing",
  "ignore" : "mutastic ignore m855c8e3c --reason=\"<why>\" samples/pricing",
  "propose" : "mutastic ignore m855c8e3c --propose --as=<equivalent|unreachable|…> --reason=\"Differs only when <case>. That can't happen because <why>.\" samples/pricing",
  "rendered" : "missed m855c8e3c: no test checks `discount()` at `total == 100`\n   at src/main/java/com/example/pricing/Pricing.java:17\n   - if (total >= 100) {\n   + if (total > 100) {\n   covered by: PricingTest#bigOrdersGetTenPercentOff, PricingTest#smallOrdersPayFullPrice; all still pass with the change\n   evidence: every test calls `discount()` away from the boundary\n   evidence: inputs seen: 50, 250 (none is 100)\n   fix: test `discount()` with `total = 100` and assert the result\n   test to add in PricingTest:\n     @Test\n     void discountAtTheTotalEdge() {\n       assertEquals(10, new Pricing().discount(100)); // today 10 · with this mutant 0\n     }\n     // check today's value is the one you intend: the test pins what the code does now\n   done when: mutastic verify m855c8e3c samples/pricing passes\n   reproduce: mutastic run --only=m855c8e3c samples/pricing\n   or propose: mutastic ignore m855c8e3c --propose --as=<equivalent|unreachable|…> --reason=\"Differs only when <case>. That can't happen because <why>.\" samples/pricing\n"
}

status is what became of it, named as the mutation-testing report schema names it: Survived is missed, Killed caught.
id names it to verify, explain and ignore, and stays the same while code around it moves. location is where it is: the file and line, the class, and the method with its JVM descriptor.
operator is the kind of change, and docs where the docs describe it; a plugin’s operator has no docs.
mutation is the change: in words in description, and in applied as the engine made it, here to the compiled instruction.
coveredBy and inputs are the evidence: the tests that reach the line, which all still pass with the change in, and the values they passed.
suggestedTest is the test that would catch it, with what the call gives today and with the mutant in.
doneWhen is the command that proves a fix, and rendered the whole miss in the agent format’s words, for a coding agent to read as it is.

The schema gives every field, its type, whether every event has it, and the values it can take.

Editions

The same events come from each edition of mutastic, Java’s and Rust’s. Every event’s edition says which wrote it, java or rust. One field is required only of the Java edition, because only it has it: run-finished’s jdk. What every edition shares says the same in its own words: toolchain is {"name": "jdk", "version": "17"} from Java and {"name": "rustc", "version": "1.91.0"} from Rust, and a mutation’s applied is the change as each engine makes it: {"level": "bytecode", "text": "IF_ICMPLT -> IF_ICMPLE"} from Java, to the compiled instruction, and {"level": "source", "text": "x < 0 -> true"} from Rust, to the source.

The Rust edition writes some fields of its own, each marked as such in the schema:

Field	In	What it is
`profile`	`run-started`	The build settings that decide what a mutant means: the profile’s name, `optLevel`, `overflowChecks` and `debugAssertions`
`unsafe`	`run-started`	Which code near `unsafe` was mutated: `skip`, `enclosing` or `all`
`location.column`	`mutant-finished`	The column, from 1
`nearUnsafe`	`mutant-finished`	Why the mutant is near unsafe code, when it is: it runs on its own, and a crash isn’t a catch
`RuntimeError`	a mutant’s `status`	The tests crashed the process where the crash may be undefined behaviour: nothing caught it, and it’s out of the score
`runtimeError`	`run-finished`	How many mutants were `RuntimeError`
`notMutated.unsafe`	`run-finished`	How many mutants weren’t made because the `unsafe` setting leaves their code out
`build`	`run-finished`	How many times the crate was compiled with mutants in, how many mutants didn’t compile, and how long reading and building the package took (`setupMs`)

In the Rust edition, location.class is the path of what holds the function and location.method is the function’s name, where the Java edition has a class’s binary name and a method’s name and JVM descriptor.

The schemas

mutastic publishes a JSON Schema (2020-12) for each format another tool reads or writes:

Schema	Describes
`/schemas/events/v1.json`	The events `--format=jsonl` writes, one per line
`/schemas/mutastic-yaml/v1.json`	A project’s `mutastic.yaml`
`/schemas/rules/v1.json`	A rules file of call operators

Each schema’s $id is the address it’s served at. mutastic’s own tests check every event their runs write against the events schema, field for field, and every mutastic.yaml and rules file they read against theirs, so a schema can’t drift from what mutastic does.

The report files follow formats of their own: mutation.json is the mutation-testing report schema’s, version 2, and mutastic’s tests check it against that schema too. See Report files.

What v1 promises

This is the events format’s part of what stays the same between releases; Compatibility has the rest: mutant ids, scores and the plugin API.

From 1.0.0, while events say mutastic.events/v1:

A field keeps its name and its type. A field every event has stays in every event, and a field only some events have stays optional, so a reader that handles it missing keeps working.
A value a field can take stays one it can take: each status, skip kind and failure origin.
Each event keeps its reason.

Any release may add:

A field, always optional.
An event, with a reason of its own.
A value a field can take, such as a status, a skip kind or a mutation’s applied.level: an engine that changes code at another level, such as llvm-ir, says so with a level of its own, and its text is opaque.

So read what you know and let the rest pass: ignore a field you don’t know, ignore an event whose reason you don’t know, and treat a value you don’t know as neither caught nor missed rather than failing. Validate against the schema as it’s published, which allows fields it doesn’t name; don’t close its objects.

Some fields hold words for people and agents to read: rendered, rule, decline, a mutation’s description and the commands offered with a missed mutant. The field stays, but its words can change from one release to the next. The order of the fields in an event isn’t part of the promise.

Before 1.0.0, a release that has to break one of these says so in its release notes.

What counts as breaking

Before 1.0.0 the schema may change; from 1.0.0 it only gains. The build compares the events schema with the last 1.0+ release’s, and fails when a change:

removes a field, or renames one, which is the same thing to a reader;
makes an optional field required, or a required one optional, including for one edition’s events only;
changes a field’s type, whether narrowing it (a number that’s no longer ever null) or widening it (a whole number that can now have a fraction);
removes a value from those a field can take;
adds, changes or removes a constraint, such as the pattern an id follows;
removes an event.

A new optional field, a new value, a new event and new descriptions pass.

How a version moves to v2

A change that would break a v1 reader comes only with a major release of mutastic, as mutastic.events/v2, with a schema of its own at /schemas/events/v2.json. Every event says which version it is in its schema field, so a reader can tell before it reads anything else. v1’s schema stays published at its address.

mutastic.yaml and rules files

mutastic reads both strictly: a key it doesn’t know is an error, so the schemas refuse one too. An editor that knows JSON Schema can then flag a mistake before a run does. With the YAML language server (VS Code’s YAML extension, for one), name the schema in the file’s first line:

# yaml-language-server: $schema=https://docs.mutastic.io/schemas/mutastic-yaml/v1.json
rules:
  - skip: com.acme.generated.**
    reason: protobuf output

mutastic reads each value in a rules file as text to the end of its line, where YAML would stop at a : or a leading backtick. Quote such a value, as the files mutastic ships do, and both read it the same.

Within v1, these formats keep every key mutastic reads, and may gain new ones.