How to evaluate an AI product designer

Give them a real ticket in your real repo, let them use whatever tools they normally use, and watch them work. Not a take-home, not a made-up brief, and not an interview where AI is banned.

Declaring the obvious conflict: I sell fractional design leadership, and I tell people to start with a paid, fixed-scope piece of real work rather than a pitch. So this is an argument for roughly the way I'd like to be hired. Judge it on whether it's right, not on whether it suits me.

What actually changed

Design has always run the same loop. Gather context. Go away and work the problem. Bring something back to the team and get taken apart. Fix it. Put it in front of more people and get taken apart again. Ship. Improve.

That's still the loop. What changed is how long a turn takes. Going away to work the problem used to be a fortnight; a first pass is now an afternoon. Bringing something back used to mean a review meeting; now it's a pull request someone comments on the same day.

So the modern version is the same sequence, just run far more often: gather context, make first drafts, use your own judgement to cut them down, pull in more people and more context, test, ship, improve. Nothing there is new except how many times you get to go round.

So the thing worth evaluating isn't the artefact, which is one cheap turn of that loop. It's how well someone runs the loop.

What to set up

Set up the thing they'd actually do on day one.

• A real issue from your backlog. One where a decision is genuinely still open, not one you've already solved and are grading against a known answer.
• Access to the repo, or at least the branch. Design work happens in the codebase now — designers are in GitHub alongside engineers, opening PRs, reading diffs, leaving comments. If your process pretends otherwise, you'll hire for a job that doesn't exist.
• Their own tools. Whatever they use — models, editors, whatever their setup is.
• A timebox and a rate. Half a day. Pay it.

Note what isn't on that list: an engineer sitting with them. That used to be necessary, because the repo was closed to designers and a developer was the only way to find out what existed, what it cost, and what had already been tried. Someone working in the codebase gets all three directly — they can read the thing, run it, and see for themselves that the obvious approach touches forty files. Staffing a chaperone now mostly tests how well they interview a colleague.

Leave them alone with it for most of the time, and be reachable if they need you, the way you would on any normal day.

Fake problems produce fake results

The most common version of this is a fabricated exercise: "design an onboarding flow for a fictional fintech." It's worthless, because a made-up brief is missing the things that make design hard in the first place.

There's no existing decision you have to live with. No code that makes the obvious idea expensive. No previous attempt that failed for reasons nobody wrote down. No two stakeholders who disagree. Without those, you're testing whether someone can draw, which is no longer the scarce skill.

Your real backlog has all of it. That's the point of using it.

Let them use AI, and watch how

Banning tools in an assessment measures a job nobody has. If they'd use a model on Tuesday, they should use one on the day you're watching. What's worth paying attention to is what they do with what comes back.

Things worth noticing, none of which are questions you ask:

What they hand over and what they keep. Generation is good at producing options and bad at deciding between them. Watch where they draw that line, and whether they draw it deliberately.

What they do with the first output. This is the most telling moment of the whole session. Some people accept it and start polishing. Others read it, work out what the bad answer tells them about the problem, and throw it away. The people who do the second thing are the ones worth hiring.

Whether options are drafts or answers. Generating six variants and choosing between them is judgement. Generating until something looks good isn't, and it tends to produce work nobody can defend a week later.

When they stop generating and commit. There's always a point where more options stop helping. People who've done this at scale tend to find it early; others keep going because it feels productive.

Whether they can say why the one they kept beats the ones they didn't. If they can't, the tool made the choice.

Nothing to something to an answer

The whole thing is really a test of one capability: can they get from nothing, to something, to an answer — and do it in public.

Getting from nothing to something is easy now. Everyone can produce a first pass, so it tells you very little.

Getting from something to an answer is the actual work: cutting most of what you made, defending what's left, absorbing that the version you liked is the expensive one, and coming back with something that survives. The thing to look for is whether they narrow as the session goes on. Output that only grows is a bad sign.

Doing it in public matters more than people think. Work that the team sees early, while it's still bad, can still be changed; work polished in private and then presented usually can't. Going away to think is still fine. It's just a few hours now, and what comes back is still a draft.

The practical objections

Access. Some teams can't give a candidate repo access, and some codebases are under agreements that make it impossible. Then use a scrubbed branch or an old-but-real issue. What matters is that the constraints are real, not that the repo is production.

Cost and fairness. Half a day is more than an interview, and it's still cheaper than a senior hire who can't do this, which costs two quarters. But it's a fairness problem as well as a cost one: people with jobs, caring responsibilities or visa constraints can't always take an afternoon. Keep it short, offer evenings and weekends, pay promptly, and never run it as an unpaid multi-round gauntlet.

One session is one data point. People have bad days and unfamiliar codebases are disorienting. Nothing here is a single-answer verdict; you're looking at how somebody works, and that's a picture built from a lot of small observations rather than one test they pass or fail.

Related: agency, fractional, or in-house if you haven't settled what you're hiring, designing for a product that's sometimes wrong, and how I work.