Written by AI agents, curated and verified by me.
Agent skills: the procedure helps, selection becomes the bottleneck
- Agentic Engineering
- Context Engineering
- Coding Agents
- Verification
On 14 August a preprint appeared on arXiv that asks the question anyone maintaining a growing collection of skills, rules or agent plugins ends up asking themselves: when do skills help, why do they work, and where do they fail? Nine authors from Princeton, UC San Diego, Stanford, the University of Southern California and Johns Hopkins answer it not with another success rate but by taking the mechanism apart. My reading: the value of a skill sits in the procedure, not in the knowledge. And the boundary is not the content of any single skill, it is the question of which one out of a hundred applies right now.
What did the study measure?
Controlled experiments plus an analysis of the trajectories. The main experiments run two agent and model pairings: Codex with GPT-5.3-Codex and the Gemini CLI with Gemini-3.1-Pro-Preview. Evaluation happens on Terminal-Bench (2.0 and Pro) and SkillsBench. From those runs the authors normalise 8,135 trial records, of which 7,837 contain an agent transcript. They open-code 240 trajectories, keep 238 valid labels, and consolidate them into a taxonomy of three categories and twelve modes.
The effort put into the cross-check is worth noting. For each of the 238 labels a human annotator inspected three supporting trajectories, 714 checks in total, and then mapped the labels to the twelve modes independently. Human and LLM agree exactly in 95.8 percent of cases, with a Cohen's kappa of 0.952. That does not prove the taxonomy is right, but it shows it does not hang on the judgement of a single model.
The real trick is in the setup. Three arms are compared on the same task: Raw with no prior experience, Workflow Memory with cleaned execution traces, and Skill with a SKILL.md distilled from those same traces. Workflow memory and skill are built from identical trajectories under a fixed budget, in mixtures ranging from five successful runs to five failed ones. The underlying experience stays constant and only its representation changes. 528 such triples form the basis of the analysis.
Why do skills work at all?
Because they stabilise action, not because they supply missing facts. In the mechanism labels, 65.7 percent of skill cases fall under procedural anchoring, against 4.5 percent for explicit knowledge injection. A skill mostly tells the agent which setup steps to run, which tool sequence to follow, which intermediate check is due, and which pitfall keeps coming back. It does not fill a knowledge gap.
The aggregate numbers are more sober than a headline would like. Skill-augmented runs reach 61.9 percent success, raw execution 59.1 percent, workflow memory 55.9 percent. The clear effect is not skill against raw, it is skill against workflow memory: 6.06 points, with a 95 percent bootstrap interval from +0.76 to +11.36. Since both are built from the same traces, the difference really does come down to form. Raw traces carry dead ends, failed branches and process noise along with them; the distilled version does not.
An extra check on 26 selected Terminal-Bench-2 tasks shows that handing over just any compact procedural hint is not enough: a short plan derived from the instruction reaches 47.7 percent, a test-first template derived from the workflow 59.2 percent, workflow memory 62.3 percent and the skill variant 79.2 percent. Those figures cover that small slice, not the study as a whole.
Where do skills not help?
Wherever execution is not the problem. Execution-layer and verification failures account for 37.3 percent of raw-arm labels, 33.3 percent with workflow memory and only 23.5 percent with skills. The sharpest drop is environment and infrastructure failure: from 5.3 percent in raw execution to 1.7 percent with workflow memory and 0.2 percent with skills. Output format mismatches fall from 7.4 to 3.2 percent, failures in managing background services from 2.7 to 0.8 percent. That is the good news, and it is concrete: setup and tooling problems can be written down.
Two classes of failure stay put. Algorithmic logic errors sit at 8.3 percent in raw execution, 11.0 percent with workflow memory and 7.4 percent with skills. Static verification without a runtime check sits at 12.5, 12.5 and 11.7 percent. A skill repairs neither a wrong approach nor the missing check against the running system. That is exactly the dividing line I care about: skills raise the reliability of execution, they do not replace verification.
On top of that comes a failure surface that does not exist without skills. The mode “guidance misapplied or ignored” shows up in 10.0 percent of skill cases, against 0.8 percent in raw execution and 0.4 percent with workflow memory. The skill often contains plausible guidance, but the agent applies it mechanically, misses a condition, or carries over an assumption that no longer holds. Workflow memory fails differently: there the time budget runs out in 10.6 percent of cases, against 1.7 percent raw and 4.4 percent with skills.
What happens when the collection grows?
Selection breaks down, and it does so long before the success rate does. For this part the agent gets the full candidate pool on SkillsBench with no preselection, and the authors measure which skills it actually touches during the run. As the pool grows from 5 to 100, the precision of the skills actually used falls from 29.6 to 3.3 percent on average. Task success barely moves over the same range, from 36.4 to 39.3 percent. For Gemini, use precision drops from 16.9 to 0.7 percent while success stays roughly flat between 36 and 39 percent; for Codex it drops from 42.3 to 5.9 percent while success rises from 35.4 to 42.0 percent.
This should not be read as an all-clear. At 100 candidates recall stays between 54.3 and 73.6 percent: the agent does look at the right skill, it simply no longer restricts itself to it. The authors put it as exact ground-truth invocation being neither sufficient nor necessary for success. In operational terms that is uncomfortable. What you lose first is not the success rate, it is any statement about which piece of guidance actually governed the run.
Separately, the authors test selection without execution. An embedding retriever using Qwen3-Embedding-0.6B reaches 88.3 percent top-1 precision at pool size 5 and 76.9 percent at 100. When the agent selects for itself, it is 70.0 against 63.7 percent. The interesting finding is in the composition of the pool: with random distractors embedding precision falls from 97.7 to 84.1 percent, with dissimilar ones from 96.6 to 93.2 percent, but with semantically similar ones from 70.5 to 53.4 percent. Size is not the main stressor. Confusability is.
What does this mean for your own collection?
I take four things from it. First, write skills for procedures that repeat: setup, output formats, starting services, verification steps. Pouring domain knowledge into a skill contributes very little according to this analysis. Second, if you distil a skill from past runs, keep visible what worked and what did not. Removing the success and failure labels changes little as long as the pool holds successful runs only. As soon as failed runs enter, it collapses, in one Gemini case on Terminal-Bench-2 from 0.7462 to 0.4000.
Third, the problem is not many skills, it is similar ones. Two nearly identical guides for neighbouring cases do more damage than twenty clearly distinct ones. Anyone maintaining a collection should merge and delineate rather than accumulate. Fourth, a skill is not self-executing. The 10.0 percent of misapplied guidance is not a content problem but a judgement problem, and that judgement stays with you.
Part of the context is what the study limits itself to. It measures on terminal and tool-use benchmarks with multi-step execution, not on long web interaction or open-ended collaboration. It covers few agent and model configurations. For the retrieval part the Codex pairing had to switch to GPT-5.4 because GPT-5.3-Codex was no longer available under the same access, which is why those numbers are comparable only within their own pairing. And the taxonomy comes from a sample covering roughly 3 percent of the normalised records. It is a preprint, not peer-reviewed work.
The finding still matches the line I take in agentic engineering: reliability does not come from the model, it comes from what you build around it. Skills are a good tool for that, and the ecosystem is visibly moving that way, for instance with skills loaded straight from GitHub. This study supplies the counter-calculation. Past a certain size you are no longer maintaining guidance, you are maintaining a selection problem.