Working test-first

One cycle, on the pricing sample, with every command and what it prints: a test from the spec, the least code that passes it, mutastic on the cycle’s lines, the tests its cards ask for, and a commit. Working test-first says why mutastic comes where it does.

The spec for this cycle: loyal customers get 5 off orders of 30 or more.

Before you start

Start from a clean tree, and commit at the end of each cycle, not before: --since=HEAD runs the mutants on the lines changed since the last commit, so the commit marks where the cycle started.

Write one failing test

// Loyal customers get 5 off orders of 30 or more.
@Test
void loyalCustomersGetFiveOff() {
  assertEquals(5, new Pricing().loyaltyCredit(50));
}

And just enough code for it to compile:

/// 5 off orders of 30 or more, for loyal customers.
public int loyaltyCredit(int total) {
  return 0;
}

The test fails, so mutastic won’t run yet: a test that fails with no mutant in would look as if it caught every mutant.

mutastic run samples/pricing

mutastic: the tests fail without any mutant: com.example.pricing.PricingTest#loyalCustomersGetFiveOff; their output is in samples/pricing/build/mutastic/worker.log

Make it pass with the least code

/// 5 off orders of 30 or more, for loyal customers.
public int loyaltyCredit(int total) {
  if (total >= 30) {
    return 5;
  }
  return 0;
}

The whole suite passes.

Run mutastic on the cycle

mutastic run samples/pricing --since=HEAD

 Missed ────────────────────────────────────────────────────────────

 ❯ src/main/java/com/example/pricing/Pricing.java  2 missed in 1 method

  ✗ MISSED  no test checks loyaltyCredit() at total == 30 · m809c9c9b
    ┌─ src/main/java/com/example/pricing/Pricing.java:33
    │
 32 │   public int loyaltyCredit(int total) {
 33 │     if (total >= 30) {
    │               ^^ >= changed to >
 34 │       return 5;
    │
    = mutant  - if (total >= 30) {
              + if (total > 30) {
    = also    ✗ no test needs total >= 30 to be false in loyaltyCredit() · md0b32a4f
              + if (true) {
    = note: every test calls loyaltyCredit() away from the boundary
            inputs seen: 50 (none is 30)
            covered by PricingTest#loyalCustomersGetFiveOff (its 1 covering test still passes)
    = help: one test for each mutant

            test loyaltyCredit() with total = 30 and assert the result · m809c9c9b
            // in PricingTest
            @Test
            void loyaltyCreditAtTheTotalEdge() {
              assertEquals(5, new Pricing().loyaltyCredit(30)); // today 5 · with m809c9c9b 0
            }
            check today's value is the one you intend: the test pins what the code does now

            add a test where total >= 30 is false, and assert what loyaltyCredit() does then · md0b32a4f
            // in PricingTest
            @Test
            void loyaltyCreditJustBelowTheTotalEdge() {
              assertEquals(0, new Pricing().loyaltyCredit(29)); // today 0 · with md0b32a4f 5
            }
            check today's value is the one you intend: the test pins what the code does now

    done when  mutastic verify m809c9c9b,md0b32a4f samples/pricing passes
    reproduce  mutastic run --only=m809c9c9b,md0b32a4f samples/pricing
    or ignore  mutastic ignore <id> --reason="<why>" samples/pricing for each of m809c9c9b, md0b32a4f

 Not covered (no test reaches these lines)
   ○ src/main/java/com/example/pricing/Pricing.java:36 loyaltyCredit() (1 mutant)
     try: assertEquals(0, new Pricing().loyaltyCredit(0));

   Classes  0 clean | 1 with misses (1)
   Mutants  ✓ 2 caught · ✗ 2 missed · ○ 1 not covered (5)
   Changed  8 lines in 1 file since HEAD
  Worst at  Pricing.loyaltyCredit() 2 ✗
 Test runs  4 (16 without coverage selection)
  Start at  17:58:44
  Duration  2.0s (setup 615ms, coverage 285ms, mutants 1.1s)
     Tests  4 on JDK 17 · 1 worker
   Reports  build/mutastic/mutation.json · build/mutastic/mutation.html
     Score  40.0%  threshold 80  · 50.0% where tests reach

  FAIL  Score below 80: 2 missed, 1 not covered.
 Next: mutastic verify <id> samples/pricing · mutastic run --only=<id> samples/pricing · mutastic explain <id> samples/pricing

The one test is at 50, so nothing checks the edge at 30, nothing checks an order below it, and no test reaches return 0 at all: the code ran ahead of its tests. The card suggests a test for each.

Write the card’s tests

Check each value against the spec before you keep it: an order of exactly 30 gets 5 off, and one of 29 gets nothing. Here both are what the spec asks for. Had one not been, the test would keep the spec’s value and fail on today’s code: that’s a bug found, and the cycle goes back to green.

@Test
void loyaltyCreditAtTheTotalEdge() {
  assertEquals(5, new Pricing().loyaltyCredit(30));
}

@Test
void loyaltyCreditJustBelowTheTotalEdge() {
  assertEquals(0, new Pricing().loyaltyCredit(29));
}

verify is the card’s red and green: it fails until a test catches each mutant, and passes now.

mutastic verify m809c9c9b,md0b32a4f samples/pricing

mutastic: m809c9c9b is caught by com.example.pricing.PricingTest#loyaltyCreditAtTheTotalEdge
mutastic: md0b32a4f is caught by com.example.pricing.PricingTest#loyaltyCreditJustBelowTheTotalEdge

A mutant you can’t catch

A mutant no test could catch, because its case can’t happen or no caller can see the change, is proposed for a person to ignore, with its reason (see Equivalent mutants). Don’t change the code so the mutant can’t exist: that hides what it found.

Refactor, run again, commit

Change the structure if it needs it, with the suite green, and run the cycle again:

mutastic run samples/pricing --since=HEAD

   Classes  1 clean (1)
   Mutants  ✓ 5 caught (5)
   Changed  8 lines in 1 file since HEAD
 Test runs  7 (30 without coverage selection)
  Start at  23:14:53
  Duration  1.8s (setup 559ms, coverage 261ms, mutants 985ms)
     Tests  6 on JDK 17 · 1 worker
   Reports  build/mutastic/mutation.json · build/mutastic/mutation.html
     Score  100.0%  threshold 80

  PASS  Every mutant was caught.

Then commit. The commit is the next cycle’s HEAD.

One card at a time

mutastic next --since=HEAD gives the cycle’s missed mutants one at a time, the one most worth a test first, in place of the whole run: see Working through misses. It leaves out the lines no test reaches, which run lists under Not covered.

Agents

A coding agent runs the same cycle through the agent format and the gate, whose one rule for it is the agent format’s rules: line: add or change tests only, and never change production code to make a mutant go away.