Skill Evaluation

Skill evaluation is the mandatory test suite every Claude Skill must pass before it earns a spot in the team library. You collect real past work with known correct outputs from previous manual reviews, run the Skill against each case, and score it on how well it matches the ground truth without adding noise. The process checks for three failure modes: missed issues that should have been flagged, correct issues that get ignored, and worst of all the invented issues that never existed in the first place. A brand audit Skill for example gets tested against five completed projects from last year where the exact inconsistencies were already documented by a senior designer. The Skill loads, produces its report, and you compare point by point. Only when it hits 100 percent on real issues and zero inventions does it ship. This turns subjective design judgment into a reliable repeatable tool the same way a design system component gets battle tested against real screens before it hits the main branch.

Skill evaluation is not slapping together three quick tests on work you just made up for the occasion. It is not a self review where you ask the model if its output looks good. It is not something you do once when the Skill is new and then never revisit even after the brand voice guide gets rewritten or a new Claude model drops. It is not optional theater for show. Teams that skip rigorous evaluation end up with three Skills that worked in testing and fourteen that drift into uselessness within six months. The evaluation is what keeps the library honest and the outputs trustworthy week after week.

The team at Vercel learned this the hard way in 2023 when they shipped an untested UX critique Skill. The first two real uses looked decent so they rolled it out. By the fourth use it started missing critical accessibility violations in keyboard navigation flows and inventing contrast issues on components that passed WCAG standards. The whole library took a credibility hit. They pulled the Skill, built a proper evaluation suite with seven test cases pulled from their 2022 redesign projects including the dashboard, the CLI interface, three marketing pages, and two mobile views. Each test case came with the original critique notes that listed exact heuristics violated using Nielsens ten plus Vercels custom progressive disclosure rules. After fixes the Skill passed the full suite with zero inventions and has been a daily driver since. Their component naming Skill followed a cleaner path. The owner built the evaluation first using naming conventions from their 2021 token system update. Five components with known good names and rationales. The Skill suggested three options per component exactly as instructed, ranked them correctly by system fit, and never once proposed a name outside the established patterns like the camelCase disasters that plagued their 2020 library. It shipped in one round and now saves the team from naming meetings that used to eat entire afternoons.

Another concrete case comes from Brainy Papers own rollout. The design system migration Skill was evaluated against three past migrations: the 2022 button overhaul that touched 47 components, the 2023 color palette expansion across both light and dark modes, and a particularly complex icon system refactor that involved 120 assets. The test suite lived in a Markdown file in the Skill folder with before and after token mappings plus screenshots of expected diff plans. When Claude updated its reasoning patterns in 2024 the evaluation immediately showed the Skill had become too aggressive suggesting migrations for stable tokens that the reference files explicitly called out as legacy safe. The owner adjusted the rules in the SKILL.md to respect legacy compatibility notes in the reference files, reran the exact same test suite, confirmed perfect alignment with zero hallucinations, and pushed the update. Without that quarterly check the team would have wasted days chasing phantom migration tasks across three concurrent product launches.

Perform skill evaluation any time the SKILL.md changes, when reference files get updated, after every major model release from Anthropic, and on a fixed quarterly schedule owned by the person listed in the OWNER file. Use a shared spreadsheet or a dedicated eval.md file that lives next to the Skill. Columns include test name, input summary, expected issues listed bullet by bullet, actual issues returned by the Skill, misses, inventions, overall result, and notes. For the copy QA Skill you test against real microcopy from shipped products like the error messages in your last two app releases at Stripe, the homepage copy from last quarters landing page tests, and the email sequences that drove your highest conversion rates in 2023. The goal is not perfection in the poetic sense but predictability on structural tasks. The Skill must behave exactly as your best designer would on brand consistency checks, UX heuristic walks, and token mappings. The PR process for the design-skills repo requires the updated eval spreadsheet attached. No eval results, no merge.

Avoid skill evaluation only for Skills that live in personal folders and never touch the shared library or client work. Never skip the step for anything destined for the team repo no matter how small the change seems. A Skill that passes three tests but fails the fourth by inventing two nonexistent brand violations will train your team to double check everything it outputs which defeats the entire purpose of building the library in the first place. The same rule applies to Skills that miss recurring issues like tone drift in marketing copy. If the evaluation does not catch it early the drift compounds and your brand starts to feel inconsistent at scale. False positives are worse than misses because they create busywork where none exists and kill adoption overnight.

A Skill without evaluation is just expensive drift waiting to poison your entire library.