Written by AI agents, curated and verified by me.
SkillEvaluator: not the skill collection, the measured contribution
- Agentic Engineering
- Verification
- Coding Agents
- NVIDIA
Most skill collections I see have grown the way a toolbox in a garage grows. Someone adds a skill because it looks plausible, writes down how a thing is done properly, and files it away. After that, nobody checks it again. Whether the agent even loads it, whether the run ends better with it than without, whether it collides with two other skills: all guesswork. On 19 August, NVIDIA open-sourced a tool that turns exactly that guess into a question you can answer. My thesis: a skill whose contribution you have not measured is an opinion in the context window.
What is NVIDIA SkillEvaluator?
SkillEvaluator is an open source tool that measures how skills affect agent performance, through static checks and through real task runs with and without each skill. Its subject is NVIDIA verified Skills: packaged, signed capability descriptors that tell an agent what an NVIDIA product does, when to invoke it, and how to call it. The sentence I like best in the piece is NVIDIA’s own: the verified part is the measurement that determines it is ready.
The post reports the first benchmark results for more than 300 verified skills across over 30 NVIDIA products. Each skill was evaluated on two independent harnesses. NVIDIA publishes plugins for Claude Code, Codex, and Cursor, and the same skills are available through Skills.sh, ClawHub, and Hermes Hub.
How is a skill evaluated?
In three tiers, before it is published. Each answers a different question, and each can run on its own.
- Tier 1, safety and structure. Static checks: schema and frontmatter validation, quality scoring, security scanning for prompt injection and data exfiltration, secret and PII detection, license checks, and script linting.
- Tier 2, distinctiveness. Embedding similarity finds duplicated guidance inside a single skill and overlapping coverage across the catalog.
- Tier 3, live evaluation. An agent runs against generated tasks, once with the skill installed and once without, each inside an isolated sandbox. The difference is what gets measured.
Tier 2 is the one that practically never happens in home-grown collections, and it targets a real problem: every skill in an environment competes for the agent’s attention, and one that loads when it is not relevant can reduce performance.
What makes the third tier a controlled comparison?
That everything except the skill stays the same. Tier 3 uses Harbor, an open-source framework for running agent evaluations in repeatable, isolated environments. SkillEvaluator handles the setup: it turns evaluation cases into tasks, runs the agents in sandboxes, collects the results, and calculates the skill’s impact. For each case, a harness runs twice, once with the verified skill and once without, each run in its own isolated sandbox with the same prompt, model, task inputs, and grading criteria. Within a harness, the only experimental variable is whether the skill is installed. The difference in scores is called Skill Lift and is reported in points.
The workflow is two commands long. skillevaluator create-eval-dataset ./my-skill --full creates evals/evals.json; each case holds an ID, a prompt, and an expected output, plus optional assertions. With --full, the dataset includes explicit, implicit, contextual, and negative cases. After you review the cases, skillevaluator tier3 evaluate ./my-skill --agents codex starts the comparison. That you are meant to review the cases first is not a formality, it is the lever: a skill can only be measured as precisely as its evaluation set describes the job.
What do the first numbers show?
All figures come from the 12 August 2026 snapshot of benchmarks.json at commit 738d79e, macro-averaged across the published skill-harness pairs. Without a skill, average scores were 46 out of 100 for Correctness, 42 for Discoverability, 39 for Effectiveness, and 43 for Efficiency. Security was the exception at 97; there the primary objective was to verify that installing a skill introduced no regression.
With the skill, the scores rise to 87 (Correctness, plus 41 points), 82 (Discoverability, plus 40), 78 (Effectiveness, plus 39), 78 (Efficiency, plus 35), and 98 (Security, plus 1). Across all dimensions that averages to plus 31 points, plus 39 excluding Security. Split by harness, Claude Code lands at plus 34 across all dimensions and plus 42 excluding Security, OpenAI Codex at plus 29 and plus 36. NVIDIA attributes the difference to the harnesses’ different default system prompts, context handling, and tool-calling implementations.
How solid are these numbers?
Less solid than the point differences suggest, and NVIDIA says so itself in a section titled “Known limitations”. Correctness, Effectiveness, and Security measure the outcome of a run. Discoverability and Efficiency also measure how the skill is used, whether the agent finds it, reads it before acting, and avoids unnecessary steps. Without the skill, those actions do not exist. The baselines sit at 42 and 43 rather than zero because of scoring components such as clean execution and correctly leaving the skill unloaded on unrelated tasks. So read those two values as evidence that the skill is activated correctly, rather than as a measure of unaided agent behaviour.
Then there is variance. Most skills were evaluated with a single attempt per task: 85 percent of the published results ran one attempt, 15 percent ran two. Live agent runs vary, so individual skill scores vary, and the post explicitly reports no confidence intervals. The scores are also not pass-probability estimates, they show higher average performance on the evaluated specialized tasks. And the obvious point has to be said: NVIDIA measured its own skills against its own products. What is interesting is therefore not how large the number is, it is that the number exists at all.
The three findings that reach beyond NVIDIA’s catalog
First: better evaluation datasets produce better skills. Teams that clearly define which tasks matter, which outputs are expected, and what is out of scope get sharper signals, and they get them before any agent runs.
Second: the product matters more than the agent. Skill Lift varies far more across products than across harnesses. Claude Code and Codex differ by about 5 points on average, while per-product Skill Lift ranges from roughly plus 2 to plus 46. The domain, the task, and the evaluation design weighed more than the choice of tool.
Third, and this is the finding that convinced me most: token savings are not automatic. SkillEvaluator tracks token usage separately from the Efficiency score. In two single-attempt examples, jetson-optimize-memory cut usage from 617,306 to 142,540 tokens (76.9 percent) and execution time from 474.9 to 220.0 seconds (53.7 percent). Conversely, cuopt-install increased usage from 25,227 to 55,582 tokens (plus 120.3 percent) and execution time from 34.0 to 41.1 seconds (plus 20.8 percent). Both skills are verified. The second one costs more than it saves, and you only know that because somebody looked.
Two pilots show this turning into a product signal. OpenClaw is piloting SkillEvaluator for official organizations on ClawHub, displaying with-skill and without-skill results in a dedicated tab, right where developers discover and adopt skills. Nous Research tested the tool in Hermes Agent with an optional advisory scan by SkillSpector in the install flow, surfacing file-line findings before installation.
What does this mean for your own skills?
The setup transfers, NVIDIA catalog or not: same prompt, same model, same inputs, same grading, once with the skill and once without, in an environment that carries nothing over between runs. That is the measurement I already considered the decisive step with PERFOPT-Bench. Whoever does not produce their own number adopts somebody else’s.
It also fits precisely what last week’s preprint on how skills work showed from the research side: what decides the run is not that a skill exists, but whether the right one is pulled at the right moment. NVIDIA calls exactly that Discoverability and checks it with the same care as the correctness of the answer.
One limit of the idea belongs here. A Security score of 98 is a measurement, not a promise. Where you really want to prevent an agent from doing something, the limit belongs on the tool, as with the domain lists for web search and web fetch, not in a score. Measuring tells you whether a skill helps. Enforcing is a different job, and you need both. That is the division of labour I mean by agentic engineering: reliability lives in the architecture, and the decision about what enters the context stays with the human. Except that this human can now justify it instead of guessing.