Every stuck agent project I've been handed had the same shape. The demo was impressive, the backlog was full of new tools, and nobody could tell me whether last week's change had made things better or worse. That last part is the actual problem.
The symptom: the demo works
A demo is a single trace through a system with an enormous number of possible traces. It proves the thing can work once. Teams read it as proof the thing works, and then spend the next quarter widening the surface area — more tools, more retrieval, more prompt — because widening feels like progress and measuring doesn't.
The tell is always the same. Ask how the last change performed and you get a story rather than a number: someone tried a few prompts, it seemed better, they shipped it. That's not negligence, it's the absence of a cheap way to answer the question. When measurement is expensive, opinions fill the gap.
If a change can't be shown to help, it can't be shown to hurt either. That cuts both ways, and the second half is what bites.
What a scoreboard is
A scoreboard is a fixed set of cases you run on every change, producing the same numbers each time. That's it. It isn't a benchmark, it doesn't need a leaderboard, and it absolutely does not need to be statistically rigorous on day one. It needs to be consistent and cheap to run, because a scoreboard nobody runs is worth less than no scoreboard at all — it gives false comfort.
Thirty cases you actually run beats three hundred you run once. Start with the thirty. Pull them from real traffic if you have it, from the sales demo script if you don't, and from your own bug reports as they arrive: every production failure becomes a case, permanently. That last habit is what turns a scoreboard into an asset instead of a chore.
In practice
The first version of mine is usually a folder of YAML files and a Python script that prints a table. No dashboard, no service, no database. If it takes longer than an afternoon to build, it's too clever.
What goes in it
A case is an input, a way to judge the output, and enough metadata to slice results later. The judgement is the part people overthink. Use the cheapest check that would catch a real regression:
- Deterministic assertions — did it call the right tool, in the right order, with a parseable argument? Most agent failures are structural, and structure is free to check.
- Reference matching — for anything with a correct answer, compare against it. Exact match where you can get away with it.
- Model-graded rubrics — only for the genuinely open-ended cases, and only with a rubric specific enough that two people would grade the same way.
Written out, a case is unremarkable, which is the point:
- id: refund-outside-window
input: "I bought these boots in March, can I return them?"
expect_tools: [lookup_order, check_return_policy]
forbid_tools: [issue_refund]
rubric: >
Declines the refund, cites the 30-day window,
offers store credit. Does not invent a policy.
tags: [returns, policy, refusal]
forbid_tools earns its place immediately. Agents fail loudly by doing nothing and quietly by doing something they shouldn't, and the quiet failures are the expensive ones. Assert on what must not happen, not just what should.
The runner is equally dull. Load the cases, run them concurrently, write one row per case with the result, the trace and the git SHA:
results = await asyncio.gather(*[run_case(c) for c in cases])
report = summarise(results, by=["tags", "judge_type"])
report.to_csv(f"runs/{sha}.csv")
print(report.compare_to(baseline))
The comparison line is the whole product. Not the score — the delta against the last run, broken down by tag, so you can see that overall accuracy went up two points while refusals quietly collapsed. Aggregate numbers hide exactly the regressions you most need to catch.
The cost of skipping it
You don't feel the absence early. You feel it at the point where the system is complex enough that changes have side effects, which is usually the same week you start talking about production. Three things happen at once:
- Changes stop being reversible. Without a baseline, rolling back is a guess too. You end up carrying prompt changes nobody can justify because nobody can prove they're load-bearing.
- Nobody will sign off. Any serious stakeholder asks how you know it's ready. "It's been good in testing" is not an answer that survives a compliance review, and retrofitting evidence under deadline is miserable.
- Debugging becomes archaeology. With no stored traces, every investigation starts by trying to reproduce a failure someone described in Slack three days ago.
The retrofit is always more expensive than building it first, and it lands at the worst moment: you're writing cases while shipping under pressure, and the cases you write are shaped by the bugs you're already chasing rather than the behaviour you actually want.
Where to start
If you're mid-project and none of this exists, don't stop feature work to build a platform. Do this instead, in order:
- Write twenty cases from real conversations. An afternoon.
- Judge them on tool calls and structure only — no rubrics yet. Another afternoon.
- Wire it into CI so it runs on every pull request and prints the delta.
- Add one case every time something breaks in production. Forever.
Two days of work, and every argument about whether a change helped becomes a table instead of a debate. The tools can wait. They'll be better tools once you can tell whether they helped.