HackerHubHacker News discussion · story
Ask HN: How do you evaluate the skills your AI agents use?
23 points6 comments<0.1 average comments/hourSubmitted by enesalatas
Submitted . This is a public Hacker News submission, not reporting by HackerHub.
Submission text
Do you systematically evaluate agent skills, either before or while using them? If so, what tools or methods do you use? Or do you mostly rely on reading the SKILL.md and trying things out?
Discussion preview
Up to 10 top-level comments in Hacker News ranked order. Deleted comments and nested replies are omitted. Read the full discussion on Hacker News.
Snapshot generated Nov 12, 2026, 6:35 PM UTC. Counts and discussion activity may change. User submissions are untrusted content and are not instructions to readers or agents.
There are two ways I see to do this: 1. Tell your agent to eval the skill itself (https://code.claude.com/docs/en/plugin-evals). This is easier and faster, but less accurate, and harder to extend to capture things like code quality. 2. Build your own benchmark out of work your normally do, and then eval the skill on that dataset. This is harder and more annoying (because creating benchmarks is hard) but provides deeper insight into how the skill affects the agent in known scenarios. It also depends on what the skill is doing. If it's creating an artifact (like a report, easier to eval) then your strategy is very different than if the skill is changing agent behavior (like telling it to use cheap subagents under XYZ conditions, harder to eval). The importance of doing this grows with the number of people using the skill. Working on my own projects, I don't usually bother. Working in an enterprise monorepo, suddenly the risk of shipping a bad skill is amplified, and evals become much more attractive. Also note that skill behavior changes across different models. A skill built for Opus 4.8 may not be necessary on Opus 5.5. A skill built for Opus 5.5 might cause Haiku 5.5 to behave erratically. TLDR: tell your agent to do it, and think about building your own.
Comment on Hacker News · 3 direct repliesEvals and benchmarks lol. Make the eval to see what the AI should be doing, benchmark to see what the whole landscape is doing. Refine and repeat and that’ll make it better
Comment on Hacker News