DAY 06 · Test
Test — Verify AI Output Before It Ships
Your coverage number is green. But it was built to catch how humans fail — and AI fails at the boundaries your coverage never touches. AI PRs carry roughly 1.7x the defects.
Your coverage number measures the wrong code at the wrong bar
You have a coverage threshold. Most teams do. 80% on the line, green check on the PR, merge. That threshold was set in a world where a human wrote the code — someone who got tired near the edges, but who also understood the domain, remembered the null case from the last incident, and felt uneasy about the weird input before writing the test for it. Your 80% bar was tuned to that person's failure modes.
AI does not have those failure modes. It nails the happy path with unnerving fluency — the main flow is clean, idiomatic, well-named. Then it misses the boundary: the empty array, the off-by-one, the null from a code path it never saw, the unauthorized state, the second concurrent request. It misses these because it's completing the most probable continuation of your prompt, and boundaries are where the probable and the correct diverge.
So here is the uncomfortable arithmetic. AI-authored pull requests carry roughly 1.7 times the defects of human ones — 10.83 issues each versus 6.45 (CodeRabbit 2025). Bugs per developer are up about 54% year over year, against 9% the year before (Cortex 2026). And your gate — built for 2023 humans — waves it all through, because it was never calibrated for code that fails at the edges while looking flawless in the middle. That is a verification debt you are accruing at machine speed.
Test is the practice that verifies — and the loop that compounds
Test is two things at once. First, it is verification calibrated to how AI actually fails: a higher bar for AI-generated code, adversarial probing of the boundaries AI skips, and blocking gates that a green happy-path result cannot bypass.
Second, it is the loop that closes back into Curate. Every defect Test catches is a signal about what your context failed to teach the AI. Feed that signal upstream — into the AGENTS.md, the specs, the patterns — and the same class of error stops recurring at the source. Without that loop, you catch the same bug forever. With it, the method compounds.
Without Test, governance is theatre. You can curate context, refine every spec, and enforce every pattern, and still ship a defect that passed functional tests while looking perfect. Test is where the method meets production reality.
The five sub-practices of Test
The Hopers vs the Test-Native Org
The uncomfortable truth
If your merge gate applies the same bar to AI code and human code — the overwhelmingly common case — you are letting roughly 70% more defects per pull request through on the AI-authored ones, and you cannot even see it, because the gate is green and the happy path is clean. AI-authored PRs carry 10.83 issues each against 6.45 for human ones (CodeRabbit 2025). Your dashboard says everything is fine. Production disagrees on a schedule you don't control.
And the volume is climbing. Bugs per developer are up 54% year over year (Cortex 2026). More AI code, more defects per unit of it, verified against a bar built for a slower, differently-failing author. That gap widens every sprint you don't recalibrate.
The teams that feel this most viscerally are your best engineers — reviewing AI PRs at 2x the old volume, sensing the boundaries aren't covered, with no gate that backs up their instinct. They are doing Test manually, in their heads, and burning out doing it.
If you answered A or B to any of those, you are somewhere on the road from The Hopers to The Spot Checkers — verifying AI code with a bar built for a different kind of author, catching the same failures on repeat, unable to prove whether AI is making you faster or just busier. The ladder runs: The Hopers → The Spot Checkers → The Partial Coverers → The Quality-Gated Org → The Test-Native Org. Most teams sit in the first two, because the tooling changed faster than the gates did.
Test closes the gap not by adding manual review — that doesn't scale to AI velocity — but by making verification structural: AI-calibrated gates that block, adversarial probing that finds the boundaries AI skips, deployment metrics that tell the truth, and a feedback loop that turns every caught defect into a permanent upgrade to your context. That last part is what makes the whole method compound.
- A - Same gate for everyone. AI code and human code pass the same coverage bar, if we have one.
- B - We've talked about treating AI code differently, but in practice it's the same threshold.
- C - AI-generated code is identified and held to a higher, blocking bar — 90%+ coverage, adversarial probing, mutation testing on critical paths.
- A - No. An incident is an incident. We don't track how the code was written.
- B - Sometimes, if someone remembers. It's not in the metrics.
- C - Yes. Every PR carries AI attribution, and our DORA metrics are segmented by AI-assisted versus not.
- A - We fix it twice. And a third time. Nothing changes upstream.
- B - We grumble about it in retro. It doesn't get written down.
- C - The rejection pattern updates our context — AGENTS.md, specs, review rules — so the AI stops making that mistake at the source.
The CRAFT Scorecard places your org on the maturity ladder and names your specific gaps, not generic ones. It's free, and it'll show you something specific about your org before your next meeting ends. Take it at craftmethod.co/scorecard/free. — Luis