Why evals are the 2026 shift
Claude Code’s plugin surface grew faster than most teams’ ability to prove a skill still works after a model update, a hook change, or a new MCP server. Stars and demos do not catch regressions.
Plugin evals (cases, graders, baseline without the plugin, JSON/HTML reports) give authors a measurable loop. That is the difference between a marketplace demo and shared team infrastructure.
What to put in the suite
Start with three to five tasks you already know the answer to — the jobs the skill claims to own. Include at least one case where the user already named the artifact, so you catch rediscovery waste.
Score behavior and evidence, not prose length. A skill that invents confident paths fails even when the HTML report looks polished.
CI without theater
Gate on a threshold against the no-plugin baseline for the behaviors you care about. Save JSON as build artifacts; keep HTML for human review on failures. Re-run after Claude Code upgrades and after any change to SKILL.md, hooks, or MCP config.
Evals do not replace a published-surface audit. They answer “does it still do the job?” AuditHQ answers “what does it cost and expose when someone installs it?” — run both.
Pair evals with a scheduled scorecard
CI catches breakages on the branch you control. Marketplace peers and your own listing still drift in public. Schedule Claude plugin audits so context cost, routing, and permissions stay visible between releases.
Treat the eval suite and the AuditHQ scorecard as one release checklist: green CI, then green (or improving) install score before you advertise the update.