Evaluation loops: telling whether a design change helped

August 28, 2026 · Alex Zapadenko

Keep a fixed set of real cases, run the product against all of them after every change, and look at the results side by side. Twenty cases in a spreadsheet is enough to start and better than anything you'll reason your way to. Without something like it, a team building on a model is tuning by anecdote and calling it iteration.

I've referenced this twice in other pieces without ever explaining it, which is a reasonable thing to be annoyed about. This is the explanation.

Why the usual method stops working

In deterministic software you can reason about whether a fix worked. You changed the validation rule, you can see the new rule, the old bug is gone by construction.

A product built on a model doesn't give you that. The same input produces different output tomorrow, so a single try tells you almost nothing — you saw one sample from a distribution and it happened to be good. Worse, improving one case routinely degrades another, and the degradation is silent because nobody was looking at that case when the change went in.

So the question "did that change help?" cannot be answered by looking at the thing you just changed. It can only be answered by looking at a set of cases you fixed in advance.

What the loop actually is

Three parts, none of which needs a platform.

A fixed set of cases. Real inputs, written down, that don't change when you change the product. Twenty is a sensible start. They should come from actual failures and actual sessions, not from imagination — a bad output you have pasted into a document is evidence, while a bad output you can picture is a reassurance.

A way to run all of them. A script, a spreadsheet with a column of prompts, or a person spending forty minutes on a Friday. The mechanism matters much less than that it covers the whole set every time, including the cases that were fine last week.

A record kept over time. The point is comparison. One run tells you the state; a series of runs tells you whether you're improving, which is the only thing anyone actually wants to know.

What a designer should be judging

Most writing about evaluation is about model accuracy — did the answer contain the right fact. That's not the design question, and a design team that borrows the engineering metric ends up measuring something it can't act on.

The useful version is: for each of these twenty outputs, what does the screen look like?

Run the set and capture the interface state, not just the text. Then ask the questions that belong to design rather than to the model. Does the layout hold when the answer is three times longer than the mock assumed, or one line? Does the citation still have somewhere to attach when the model returns nothing to cite? Does the empty state say something useful, or does it say "no results"? When the answer is wrong — and in twenty cases some will be — can the user tell, and can they get back?

That's a design evaluation. It happens to require running the model, but what it's grading is whether the interface survives the distribution rather than whether the model is clever.

Judge the thing you care about

Two failure modes here, and they pull in opposite directions.

The first is grading a proxy because it's easy to grade. "Did the response mention the policy number" is checkable and almost never the thing that mattered. If what you care about is whether someone could act on the answer, that's what the rubric should say, even though it needs a human read.

The second is a rubric so vague that two people score the same output differently. The fix isn't more precision in the wording, it's grading a few together until the disagreements surface, then writing down what the disagreement was about. That written note is worth more than the score.

For most teams, a three-point scale is enough: it did the job, it half did the job, it failed. Finer scales invite arguments about whether something is a 6 or a 7, which is time spent on the instrument rather than the product.

Where it goes wrong

The set goes stale. Twenty cases collected in March describe March's product. Add cases as new failures appear, and retire ones that stopped being interesting — but keep a core that never changes, or you lose the comparison that made the whole thing worth doing.

The set gets optimised against. If everyone can see the twenty cases, work drifts toward passing those twenty. Keeping a handful of cases out of the shared set and only running them occasionally is a cheap defence.

It becomes ceremony. A loop nobody looks at is worse than no loop, because it produces the feeling of rigour without the substance. If two consecutive runs change nothing about what anyone does next, either the cases are wrong or the loop should be retired.

It can't see novel failures. A fixed set is a memory of problems you already know about. It will never surface the thing nobody thought to write down, so it belongs alongside real user sessions rather than replacing them.

The smallest version worth building

A document with twenty inputs. A column for what the product produced. A column for a screenshot. A three-point score and a sentence of reasoning. Re-run it after anything meaningful changes and keep the old runs.

That's an afternoon of setup, and it moves a team from "this feels better" to "eleven cases improved, three got worse, and here's the screenshot of one that got worse" — which is a conversation you can actually have.