LLM as Judge

LLM as judge is the middle layer of the eval stack where a model scores thousands of design candidates against a detailed rubric in seconds. You render each candidate to an image or component, send it to Claude or GPT with strict instructions, and get back structured output. The model does not decide what ships. It filters. It surfaces the best fifty from ten thousand. This layer exists because human review cannot scale to AI production volumes yet the squishy parts of craft and brand still need judging.

It is not just prompting a model and hoping for the best. Random chat feedback produces random results. It is not automation of taste either. The LLM handles mechanical parts of quality. Final taste calls between strong options stay human. A common confusion is believing the model must be perfect. It does not need to be. It needs to be consistent and cheap so the expensive human layer sees only the right work.

Anthropic shipped eval frameworks in 2023 that ML teams immediately adopted. Design teams followed in 2025 once v0 and Claude started generating full interfaces. One team ran five hundred AI drafted product descriptions through a custom Claude rubric. It surfaced the thirty worth human eyes in under two minutes for a few dollars. Linear uses similar scoring for their writing. Stripe applies it to component quality. The output is JSON with per-criterion scores, reasons, and a pass flag. Sort by total score and the queue orders itself.

The prompt matters more than the model. Feed it the exact rubric, examples of good and bad output, and a JSON schema. Ask for one-line reasons per criterion. The model becomes repeatable. Version the prompt like code. Test it on last quarter's shipped work. Tune on failures. This is how a brand encodes itself so AI can police its own output.

Use LLM as judge when you cross roughly two hundred candidates per brief. It pays for itself the moment senior designers waste time on obvious failures the model could have caught. It does not make sense for tiny projects with five options or highly subjective exploratory work where the rubric cannot be written clearly. The tradeoff is that the model inherits any biases in your rubric. Garbage criteria produce garbage scores. Yet the leverage is massive. One senior eye now runs against ten thousand candidates instead of fifty.

Most teams should start with a custom Claude rubric instead of the full Anthropic eval framework. The docs skew toward ML engineers. Designers need something simpler that owns the brand definition. Run it, read the reasons, fix the rubric, run it again. The loop improves fast.

Never let the LLM make the final call on taste. That layer stays human so the moat remains. The judge simply makes the human hour ten times more valuable by clearing the junk first.

The role shift is clear. Designers who once made assets now write and tune the prompts that judge them. The best ones treat the rubric as living brand code.

Wire this layer after your linting and visual diff are solid. The pyramid only works when cheaper layers kill the obvious garbage first.

LLM as judge turns brand opinion into executable code that scales with AI output.