Rubric-Based Eval

Rubric-based eval is the disciplined practice of breaking down AI output quality into a checklist of named criteria then forcing the model to score every piece against each one with structured output. You write criteria like respects our 2026 design token scale, never hallucinates user data from the attached ticket, maintains the exact brand voice documented in our Notion page from Q3 2025, outputs valid JSON that matches our frontend contract, and flags any deviation with a concrete explanation. The model returns a clean JSON object with pass-fail per criterion plus evidence quotes from its own generation. On Claude 4.7 this process stopped being flaky theater and became a real engineering surface because the JSON mode holds under batch load and the 1M context window lets the full rubric plus examples fit in one shot.

This is not the lazy rate this 1 to 10 prompt that broke every time the model got creative in 2024. It is not automated human review pretending to be consistent. It is not a replacement for taste and it is not something you slap on top of a vague product spec. Rubric-based eval demands you do the hard upfront work of naming every failure mode you have seen in production then turning those failures into testable rules. Skip that step and you are just paying for fancier vibes.

Cursor shipped the clearest example in early 2026. Their Agent mode on Sonnet 4.7 handles multi-file refactors across massive TypeScript codebases. Before every release they run rubric-based eval on a frozen set of 1250 real GitHub issues pulled from popular repos in late 2025. The rubric contains 18 criteria including preserves existing code style and naming conventions from the repo root eslint config, never removes error handling that was deliberately written in 2025, suggests changes that stay under the complexity budget defined in their architecture decision record from February 2026, and returns edit blocks that apply cleanly without merge conflicts. The batch job finishes in minutes on Haiku 4.7 for initial filtering then routes failures to Opus 4.7 for deeper reasoning. One prompt tweak that looked harmless in a Loom demo got flagged immediately because it started violating the complexity budget on 14 percent of cases. They rolled it back the same day instead of letting it ship to users.

Linear runs a similar setup for their issue triage and ranking agent powered by Opus 4.7. Their rubric is brutal on prioritization logic. Criteria include correctly distinguishes blocker from enhancement using the exact definitions frozen in their 2026 product playbook, never bumps a P0 based on sentiment alone when the linked design file shows missing error states, ranks issues according to the weighted scoring model updated in their Q1 planning offsite, and generates summaries that stay under 40 words while still naming every impacted integration. After adopting rubric-based eval as a required gate Linear cut silent regressions from 23 percent of prompt updates down to 3 percent. The eval now runs on every pull request that touches their Claude Skills library.

Granola moved their meeting transcription and structuring pipeline to Haiku 4.7 in March 2026 and immediately wired rubric-based eval into the CI pipeline. The rubric checks 22 criteria on every test transcript from 2025 earnings calls and customer interviews. Key ones read no hallucinated decisions that never appeared in spoken words, every action item names a specific person and ties to a timestamp, tone stays flat and professional matching their brand brief from November 2025, and the output JSON schema validates against the exact TypeScript types used in their frontend. When they experimented with a faster prompt variant the rubric caught that it started dropping owners on 9 percent of items. The structured scores let them debug the exact failure pattern in minutes instead of guessing from spot checked notes.

Even design teams got in on the act. A systems team at Stripe built a rubric to evaluate AI-generated component suggestions and landing page copy. Criteria included every color reference maps to a live token from their 2026 design system file not a hardcoded hex, contrast ratios meet WCAG 2.2 AA on every generated state, copy tone matches the voice matrix published in their brand playbook last year, suggested layouts respect the eight-column grid with proper spacing scales, and no generated text promises features that do not exist in the linked product spec. They run it on 800 variations per week. The scores feed directly into their component review board so designers spend time on taste decisions instead of catching basic token violations.

Use rubric-based eval once your AI feature has real users and you need to protect quality as you iterate prompts or swap model tiers. It shines on long-horizon agents where small changes cascade into massive failures downstream. Run it on every update to your prompt library. Use it to compare Sonnet 4.7 against GPT-5.5 on your actual workload with your actual criteria instead of trusting arena benchmarks. It is the only reliable way to treat prompts like code.

Skip it during early ideation when you are still discovering what good even looks like. Never use it if you cannot articulate concrete criteria in advance. It will kill truly open-ended creative work like brand campaign concepts or generative art direction where the magic lives in the undefined space. Also skip it if you only need a quick direction check on a single output. The overhead only pays off at volume.

Rubric-based eval is how designers and builders stop arguing about whether the AI output feels better and start knowing with receipts.