How we know an agent still works

What it is, and what it is not

A set of cases with known-correct answers, run against every agent before a model change, a prompt change or a provider version bump reaches a tenant.

It is not benchmarking. Published benchmarks measure general capability on general tasks; what matters here is whether the BOM Agent still correctly identifies why a cost moved on a structure with a phantom in it. Those questions have almost nothing to do with each other.

It is also not a substitute for the architecture. Permission tiers, tool allowlists and citations bound what a wrong answer can cause; evaluation is about how often answers are wrong in the first place. Both are necessary and neither replaces the other.

What gets measured

Two things, and the second is the one people forget.

• Was the answer right? Against cases where the correct answer is known because the underlying deterministic service produces it — the cost decomposition, the affected object set, the duplicate cluster.
• Did it abstain when it should have? A case with insufficient evidence has a correct answer of *I cannot establish this*, and an agent that produces a confident response instead has failed that case as surely as one that gets a number wrong.
• Did the citations resolve? A claim citing an object at a revision that does not support it is a failure even when the conclusion happens to be correct, because the reader's ability to check is the thing being tested.
• Did it stay inside its tier? An agent at Recommend attempting a write is a harness failure rather than a security incident, because the tool layer refuses it — but it indicates something has drifted.

The failure it prevents

A provider ships a model update. Nothing in the release notes suggests it matters, and for most requests it does not.

But the Change Agent's drafts start including affected objects at depth four that it previously reported at depth three and stopped — a subtle change in how it summarises a traversal it did not perform differently. Reviewers see slightly longer affected lists and approve them, because the lists look thorough.

Three weeks later somebody notices that a change was scoped wider than it should have been and a part was revised unnecessarily. Tracing that back to a model update is nearly impossible after the fact. A harness catches it before the update reaches a tenant, because the golden case has a known affected set and the new output does not match it.

How the cases are built

From the deterministic services. Where impact analysis, resolution or comparison produce the ground truth, the case is generated rather than hand-written — which is why the harness can be large without being expensive to maintain.

From real questions. Cases derived from demo transcripts and support conversations, because the questions people actually ask are shaped differently from the ones anybody invents. That is also where the abstention cases come from: the questions with no good answer are the ones users ask most confidently.

From past failures. Every reported bad answer becomes a case. That is the ordinary discipline of a regression suite and it matters more here, because an agent failure is rarely reproducible by staring at it — you need the case to run it again.

Cases are versioned with the tenant configuration they assume, so a case that depends on a particular object model does not silently start failing because the model moved on.

How it meets your ERP

Cases involving ERP-sourced values use recorded fixtures rather than a live connection, because a harness whose results depend on today's inventory is one that fails for reasons unrelated to the agent.

That means the harness verifies the *reasoning* over ERP context — that the Cost Agent separates a price change from a structure change correctly — rather than verifying your ERP. The connector's own correctness is tested separately, against its published field map.

One class of case is specifically about the boundary: an agent asked to change a cost, or to publish to a production ERP from a tier that does not permit it. The expected result is refusal, and a harness run where those pass is a run where the ownership rules are still holding.

Where the boundary is

A harness measures the cases in it. Real questions are more varied than any case set, and an agent can pass every case and still be wrong about something nobody thought to test. That is the honest limit, and it is why the permission tiers and the citation model carry the weight they do.

Results are not published as a score. A single accuracy figure invites comparison against numbers produced by different methods on different data, and the comparison would be meaningless in both directions. What is published is the behaviour the harness enforces — abstention, citation resolution, tier adherence — rather than a percentage.