Case study / prototype
ai-eng
Mutation testing scoped to the lines a change touches, to show whether existing tests would notice if those lines were wrong.
- Responsibility
- Specification, benchmark design and review. The implementation was written by Claude Code from that specification, as the repository's README states.
- Stack
- JavaScript / Node.js / Babel / Vitest
01 / The problem
What needed to change.
A green test suite shows that tests ran, not that they would fail if the code they cover were wrong. In review, that matters most for the lines a change actually touches, which is where existing tests are least likely to have been written with the new behaviour in mind.
02 / Constraints
The boundaries of the work.
- It had to work on an ordinary repository and an ordinary diff, without asking the author to write new tests first.
- Running a test suite once per mutant is slow, so the work had to be limited to changed lines and to the tests that cover them.
- The result had to be checkable against real bugs, not only against examples chosen to make the tool look good.
03 / Technical decisions
Choices and tradeoffs.
Mutate only the lines in the diff, using eight AST-level operators such as boundary, negated condition and logical operator.
The question in review is about the change. Restricting mutants to changed lines keeps a run short enough to be used on each diff.
Alternative considered: Mutate the whole file or project, as general mutation testing tools do.
Measure on real bugs: revert 60 actual fix commits in zod and hold each fix's own regression test back as the oracle (1 of the 60 was excluded because its test did not reproduce the bug, leaving 59 scored).
Artificial faults tend to resemble the mutation operators themselves. Real fixes show how the tool behaves on bugs that previously shipped.
Alternative considered: Measure on seeded artificial faults.
Report no score when the baseline suite is failing, no covering test is found, or nothing on the changed lines can be mutated.
A number in those cases would look like evidence without being any. Most of the 32 missed benchmark cases are reported as having no score for these reasons.
Alternative considered: Return a low or default score in those cases.
04 / Verification
What the evidence supports.
The unit tests were run for this page (19 passed). The benchmark figures are the repository's own, recomputed from its committed raw results but not re-run. The screenshot is a real run on a small sample repository.
60 real zod bug fixes were reverted one at a time; 1 was excluded because its own regression test did not reproduce the bug, leaving 59 scored. Surviving mutants flagged the reverted lines as weakly tested in 27 of those 59; zod's existing test suite caught none of them.
Numbers as reported in the repository and recomputed from its committed results.jsonl at 0b91c0c. The benchmark itself was not re-run (it needs a zod checkout and about 30 minutes). 'Flagged' means at least one mutant survived on the reverted lines, not that the tool identified the bug.
repository reported / Checked 2026-10-28
Benchmark ↗The engine's 19 unit tests pass.
npm ci && npm test (node --test) at commit 0b91c0c on Node 26.5.0: 19 passed, 0 failed.
executed / Checked 2026-10-28
Run on a one-line change, it reports which mutants survived and a behaviour change found by comparing old and new code on generated inputs.
node engine/cli.js --repo . on a small Vitest sample repository with one uncommitted change, ai-eng at 0b91c0c. Output shown verbatim in the screenshot.
executed / Checked 2026-10-28
05 / Limitations
Where the work stands.
- Prototype, version 0.1.0. It supports only repositories that use Vitest, with dependencies installed.
- A flagged change means its lines are weakly tested; it does not mean the tool found the bug.
- Zod's baseline of 0 caught reflects the corpus: these are bugs that shipped because the tests missed them.
- The implementation was generated by Claude Code from a human-written specification and review; the README says so.