Every AI-for-testing demo I have sat through follows the same script. Someone points a model at a requirements doc, it emits forty test cases, and the room nods. It's a good demo. It also solves a problem I've never actually had.

Writing tests was never the bottleneck. I've worked on suites where the tests wrote themselves out of habit, one per bug, one per ticket, until there were thousands of them running every night. The bottleneck showed up the next morning, when sixty of them were red and someone had to decide what that meant.

That decision is the job. And it's the part almost nobody is automating well.

The morning triage tax

Run the arithmetic on a mid-sized suite. Say two thousand tests, nightly, with a three percent failure rate. That's sixty red tests waiting when the team logs on. In my experience, four out of five of those aren't defects at all. They are a stale fixture, a timing window that closed, a device that did not finish rebooting, a data seed that drifted, a network hiccup in the middle of a retry.

(Illustrative numbers. If the failure rate looks low to you, you have a bigger problem than this article can fix.)

Someone still has to open all sixty. Ten minutes each, if the logs are decent. That's a full engineer-day, every day, spent sorting noise from signal. Teams call it triage and treat it like weather, something that happens to them. It's the most automatable workload in the entire pipeline, and it sits there untouched while everyone builds test generators.

Hardware is where this gets ugly

If your tests run against pure software, you can at least assume the machine underneath you is honest. I've spent a good chunk of my career on integration testing where the system under test includes physical hardware, and that assumption dies on day one.

A single red test on a device rig has, roughly, four candidate explanations:

1. The code regressed.
2. The firmware on the device changed, and the test is fine.
3. The rig itself drifted. A cable, a calibration, a sensor, a thermal condition.
4. The test is racing something physical and lost this time.

Only the first one is a bug. The other three are why nobody trusts the suite by month six, and why teams start rerunning failures until they go green. That's the moment the suite stops being a safety net and becomes a ritual.

A stack trace can't distinguish between those four. Neither can an LLM staring at a stack trace. Which brings me to the thing I actually want to argue.

The agent is the last thing you build

Every diagnostic agent I've seen fail, failed for the same reason. The pipeline underneath it had already thrown away the evidence.

Typical CI keeps a stack trace, a truncated log, and a pass/fail bit, with seven-day retention. The device serial output, the screenshot at failure, the firmware version on the rig, the calibration timestamp, the last ten runs of that same test, the data seed hash: gone, or never captured, or sitting in four different systems with no shared key.

You can't reason your way out of missing evidence. If a human engineer couldn't diagnose the failure from what your CI stores, a model can't either. It will just sound more sure of itself.

So the first thing to build isn't the agent. It's the failure bundle: one object, one ID, everything a diagnosis could possibly need.

failure_bundle:

  run_id: nightly_1187

  test_id: scan_pipeline.calibration_drift

  status: failed

  code_diff_since_last_pass: [commit_a1b, commit_c4d]

  firmware_version: 3.2.1        # changed since last pass

  rig_id: rig_04

  rig_last_calibrated: 6 days ago

  history_last_20_runs: [P,P,P,F,P,P,F,P,...]   # 3 failures, all on rig_04

  sibling_failures_this_run: [scan_pipeline.focus, scan_pipeline.align]

  artifacts: [serial_log, screenshot, host_metrics, data_seed_hash]

Look at what that bundle already tells you before any model touches it. The firmware moved. The failures cluster on one rig. Two sibling tests in the same subsystem went red together. A human glances at that and says “rig four, probably the firmware bump” in about eight seconds.

The agent's job is to do those eight seconds sixty times over before anyone wakes up, and to say out loud which evidence it leaned on.

Make the output routable, not readable

The second mistake is asking the agent for a paragraph. Prose gets read once and evaporates. You want a verdict a router can act on.

triage_verdict:

  failure_id: f_9931

  category: environment          # regression | environment | flake | unknown

  suspected_cause: rig_calibration_drift

  supporting_evidence: [rig_id, rig_last_calibrated, history_last_20_runs]

  contradicting_evidence: [firmware_version_changed]

  recommended_route: infra_queue

  blocking: false

Four categories, one route each, and both sides of the argument written down. Regression blocks the merge and pages the owning team. Environment goes to the infra queue with the rig ID attached. Flake gets logged against the test's stability score, and when that score crosses a line the test gets a bug filed against it instead of a silent rerun. Unknown goes to a human. Unknown is a perfectly respectable answer.

That contradicting_evidence field earns its keep. It's the only one that tells a reviewer what the agent had to talk itself out of, and it's the first place I look when a verdict turns out wrong.

Do not let it be confident on its own behalf

An agent writing “high confidence this is environmental” isn’t producing a probability. It’s producing a phrase. And the phrasing tends to get more assertive as the evidence gets thinner, which is exactly backwards.

If a number is going to gate an action, get the number from something you can calibrate. Score the assembled evidence with a small supervised model trained on your own labeled triage history, and let that score decide whether a verdict is auto-routed or handed to a person. The agent gathers and reasons. A boring classifier decides how much to trust it. Keep those jobs in separate processes, because once they merge you lose the ability to tell a well-argued mistake from a correct call.

One rule I'd write down before anyone builds this: the agent doesn't close the merge gate on its own. It routes, annotates, recommends. A human confirms anything that blocks a release or quarantines a test. The failure mode of an over-eager auto-quarantine is a real regression silently marked as flake, and that bug ships.

The loop that pays for itself

Here's where this stops being a triage assistant and starts being infrastructure.

Every verdict, once a human confirms or corrects it, is a labeled example. Bundle plus category plus correction. Feed those back and a plain classifier picks up your common failure signatures quickly, because most triage is repetitive. Rig four drifting isn't a novel insight the third time it happens.

Six months in, the shape you want is a cheap model handling the recurring signatures, the agent working only the unfamiliar ones, and humans seeing what's left. If the agent is still reasoning from scratch about rig calibration drift a year after it first diagnosed it, the loop is broken and you're paying agent prices for classifier work.

Measure the thing you actually care about

Not “agent accuracy.” That number is easy to game and tells you almost nothing.

Track mean time to triage, because that's what the team feels every morning. Track the share of failures no human ever has to open, because that's what you're actually buying. Then track the metric everyone avoids: how many real regressions got filed as flake. Sample the flake bucket, have a person check it, publish the number. A triage system that never misfiles a regression is probably too timid to be worth running. One that misfiles them quietly is worse than nothing, and you won't find out until a customer does.

The part I keep repeating in meetings

The generative demo is seductive because it produces artifacts you can see. Tests appear on a screen. It looks like work.

A diagnostic system produces the absence of work, and nobody applauds that. Sixty red tests become eight that need a human. The other fifty-two get sorted, routed, and filed before anyone opens a laptop. No demo moment. Just a team that stopped losing its mornings.

I know which one I'd rather ship.