Evaluation loops

Tags: evals, measurement, process

Keep a fixed set of real cases, run the product against all of them after every change, and look at the results side by side. Twenty cases in a spreadsheet is enough to start and better than anything you'll reason your way to. Without something like it, a team building on a model is tuning by anecdote and calling it iteration.

Why the usual method stops working

In deterministic software you can reason about whether a fix worked. You changed the validation rule, you can see the new rule, the old bug is gone by construction.

A product built on a model doesn't give you that. The same input produces different output tomorrow, so a single try tells you almost nothing — you saw one sample from a distribution and it happened to be good. Worse, improving one case routinely degrades another, and the degradation is silent because nobody was looking at that case when the change went in.

So "did that change help?" cannot be answered by looking at the thing you just changed. It can only be answered by looking at a set of cases you fixed in advance.

What the loop actually is

Three parts, none of which needs a platform.

• A fixed set of cases. Real inputs, written down, that don't change when you change the product. Twenty is a sensible start. They should come from actual failures and actual sessions — a bad output you have pasted into a document is evidence; one you can picture is a reassurance.
• A way to run all of them. A script, a spreadsheet with a column of prompts, or a person spending forty minutes on a Friday. The mechanism matters much less than covering the whole set every time, including the cases that were fine last week.
• A record kept over time. The point is comparison. One run tells you the state; a series tells you whether you're improving, which is the only thing anyone actually wants to know.

What a designer should be judging

Most writing about evaluation is about model accuracy — did the answer contain the right fact. That is not the design question, and a design team that borrows the engineering metric ends up measuring something it can't act on.

The useful version is: for each of these twenty outputs, what does the screen look like?

Run the set and capture the interface state, not just the text. Then ask the questions that belong to design:

• Does the layout hold when the answer is three times longer than the mock assumed, or one line?
• Does the citation still have somewhere to attach when the model returns nothing to cite?
• Does the empty state say something useful, or does it say "no results"?
• When the answer is wrong — and in twenty cases some will be — can the user tell, and can they get back?

That's a design evaluation. It happens to require running the model, but what it grades is whether the interface survives the distribution.

Judge the thing you care about

Two failure modes, pulling in opposite directions.

Grading a proxy because it's easy to grade. "Did the response mention the policy number" is checkable and almost never the thing that mattered. If what you care about is whether someone could act on the answer, that's what the rubric should say, even though it needs a human read.

A rubric so vague two people score the same output differently. The fix isn't more precision in the wording — it's grading a few together until the disagreements surface, then writing down what the disagreement was about. That written note is worth more than the score.

A three-point scale is enough for most teams: it did the job, it half did the job, it failed. Finer scales invite arguments about whether something is a 6 or a 7, which is time spent on the instrument rather than the product.

Grounded in

The smallest version worth building, and the four ways it goes wrong in practice — written because the loop had been referenced twice in other pieces without ever being explained.

Anti-patterns

• The set goes stale. Twenty cases collected in March describe March's product. Add cases as new failures appear and retire ones that stopped being interesting — but keep a core that never changes, or you lose the comparison that made it worth doing.
• The set gets optimised against. If everyone can see the twenty cases, work drifts toward passing those twenty. Keeping a handful out of the shared set is a cheap defence.
• It becomes ceremony. A loop nobody looks at is worse than no loop: it produces the feeling of rigour without the substance. If two consecutive runs change nothing about what anyone does next, either the cases are wrong or the loop should be retired.
• Treating it as complete coverage. A fixed set is a memory of problems you already know about. It will never surface the thing nobody thought to write down, so it belongs alongside real user sessions rather than replacing them.

The smallest version worth building

A document with twenty inputs. A column for what the product produced. A column for a screenshot. A three-point score and a sentence of reasoning. Re-run it after anything meaningful changes, and keep the old runs.

That is an afternoon of setup, and it moves a team from "this feels better" to "eleven cases improved, three got worse, and here's the screenshot of one that got worse" — which is a conversation you can actually have.

Related patterns

• Design the failure state first — the loop is how you find out the failure design works.
• Mark what the product vouches for — what the screen does with a claim is one of the things worth scoring.

The full argument: Evaluation loops: telling whether a design change helped.