← THE COURSEDAY 06 OF 07

DAY 06 · Test

Test — Verify AI Output Before It Ships

Your coverage number is green. But it was built to catch how humans fail — and AI fails at the boundaries your coverage never touches. AI PRs carry roughly 1.7x the defects.

Your coverage number measures the wrong code at the wrong bar

You have a coverage threshold. Most teams do. 80% on the line, green check on the PR, merge. That threshold was set in a world where a human wrote the code — someone who got tired near the edges, but who also understood the domain, remembered the null case from the last incident, and felt uneasy about the weird input before writing the test for it. Your 80% bar was tuned to that person's failure modes.

AI does not have those failure modes. It nails the happy path with unnerving fluency — the main flow is clean, idiomatic, well-named. Then it misses the boundary: the empty array, the off-by-one, the null from a code path it never saw, the unauthorized state, the second concurrent request. It misses these because it's completing the most probable continuation of your prompt, and boundaries are where the probable and the correct diverge.

So here is the uncomfortable arithmetic. AI-authored pull requests carry roughly 1.7 times the defects of human ones — 10.83 issues each versus 6.45 (CodeRabbit 2025). Bugs per developer are up about 54% year over year, against 9% the year before (Cortex 2026). And your gate — built for 2023 humans — waves it all through, because it was never calibrated for code that fails at the edges while looking flawless in the middle. That is a verification debt you are accruing at machine speed.

Test is the practice that verifies — and the loop that compounds

Test is two things at once. First, it is verification calibrated to how AI actually fails: a higher bar for AI-generated code, adversarial probing of the boundaries AI skips, and blocking gates that a green happy-path result cannot bypass.

Second, it is the loop that closes back into Curate. Every defect Test catches is a signal about what your context failed to teach the AI. Feed that signal upstream — into the AGENTS.md, the specs, the patterns — and the same class of error stops recurring at the source. Without that loop, you catch the same bug forever. With it, the method compounds.

Without Test, governance is theatre. You can curate context, refine every spec, and enforce every pattern, and still ship a defect that passed functional tests while looking perfect. Test is where the method meets production reality.

The five sub-practices of Test

T1
Generate Tests with AI, Hold AI Code to a Higher Bar
Use AI to generate the tests — unit, integration, edge-case, property-based — but don't trust coverage as your quality signal. Add mutation testing on critical paths: it deliberately breaks the code and checks whether your tests notice. Set the threshold for AI-generated code higher than for human code — 90%+, not the shared 80%.
T2
Validate Edge Cases Adversarially
Run adversarial test suites that probe AI code for its known failure signatures: off-by-one, null handling, forbidden states, unauthorized actions, domain boundaries. Feed historical bug data and AI failure patterns into the generator so the probing gets sharper over time. Attack the code the way production will, before production does.
T3
Gate Quality, Calibrated Higher for AI Code
Make the gate blocking, not advisory — no bypass without explicit approval. Identify AI-generated code and subject it to stricter thresholds: higher coverage, lower complexity limits, mandatory documentation, pattern adherence. Apiiro found AI code delivering 4x the velocity and 10x the vulnerabilities — privilege-escalation paths up 322%. Without a gate tuned for that, velocity is a liability multiplier.
T4
Measure Deployment Quality with AI Attribution
Track DORA-aligned metrics — deploy frequency, change-failure rate, MTTR — and segment them by AI-assisted versus not. Cortex 2026 found AI delivered 20% more PRs but 23.5% more incidents per PR and a 30% higher change-failure rate. You cannot manage what you cannot attribute.
T5
Close the Loop Back to Curate
Track acceptance and rework rates on AI output. When a rejection pattern repeats, update the context that produced it: AGENTS.md anti-patterns, spec templates, review rules. The best teams make this automatic — a production incident involving AI code triggers a new test case and a context update, so the same failure cannot recur. A self-healing development ecosystem.

The Hopers vs the Test-Native Org

The Hopers
AI code coverage: whatever the dev added, if anything
Edge cases: discovered when users hit them
Quality gate: none, or same bar as humans
Deployment quality: ship and hope
Feedback loop: same mistakes recur indefinitely
Net effect of AI: faster delivery, more incidents
The Test-Native Org
AI code coverage: 90%+, mutation-validated
Edge cases: adversarially probed before merge
Quality gate: blocking, calibrated higher for AI code
Deployment quality: DORA metrics, AI-attributed
Feedback loop: every failure improves context automatically
Net effect of AI: faster delivery, fewer incidents, lower MTTR

The uncomfortable truth

If your merge gate applies the same bar to AI code and human code — the overwhelmingly common case — you are letting roughly 70% more defects per pull request through on the AI-authored ones, and you cannot even see it, because the gate is green and the happy path is clean. AI-authored PRs carry 10.83 issues each against 6.45 for human ones (CodeRabbit 2025). Your dashboard says everything is fine. Production disagrees on a schedule you don't control.

And the volume is climbing. Bugs per developer are up 54% year over year (Cortex 2026). More AI code, more defects per unit of it, verified against a bar built for a slower, differently-failing author. That gap widens every sprint you don't recalibrate.

The teams that feel this most viscerally are your best engineers — reviewing AI PRs at 2x the old volume, sensing the boundaries aren't covered, with no gate that backs up their instinct. They are doing Test manually, in their heads, and burning out doing it.

If you answered A or B to any of those, you are somewhere on the road from The Hopers to The Spot Checkers — verifying AI code with a bar built for a different kind of author, catching the same failures on repeat, unable to prove whether AI is making you faster or just busier. The ladder runs: The Hopers → The Spot Checkers → The Partial Coverers → The Quality-Gated Org → The Test-Native Org. Most teams sit in the first two, because the tooling changed faster than the gates did.

Test closes the gap not by adding manual review — that doesn't scale to AI velocity — but by making verification structural: AI-calibrated gates that block, adversarial probing that finds the boundaries AI skips, deployment metrics that tell the truth, and a feedback loop that turns every caught defect into a permanent upgrade to your context. That last part is what makes the whole method compound.

SELF-ASSESSMENT
01
The Calibration Test — Is your merge gate calibrated for AI velocity, or for 2023 humans?
  • A - Same gate for everyone. AI code and human code pass the same coverage bar, if we have one.
  • B - We've talked about treating AI code differently, but in practice it's the same threshold.
  • C - AI-generated code is identified and held to a higher, blocking bar — 90%+ coverage, adversarial probing, mutation testing on critical paths.
02
The Attribution Test — When a production incident happens, can you tell whether AI-assisted code caused it?
  • A - No. An incident is an incident. We don't track how the code was written.
  • B - Sometimes, if someone remembers. It's not in the metrics.
  • C - Yes. Every PR carries AI attribution, and our DORA metrics are segmented by AI-assisted versus not.
03
The Loop Test — When AI produces the same class of bug twice, what happens?
  • A - We fix it twice. And a third time. Nothing changes upstream.
  • B - We grumble about it in retro. It doesn't get written down.
  • C - The rejection pattern updates our context — AGENTS.md, specs, review rules — so the AI stops making that mistake at the source.

The CRAFT Scorecard places your org on the maturity ladder and names your specific gaps, not generic ones. It's free, and it'll show you something specific about your org before your next meeting ends. Take it at craftmethod.co/scorecard/free. — Luis