Skills That Scan Clean Still Fail at Runtime: NVIDIA’s Skill Lift

Many skill libraries use a review gate that looks like software lint: scan SKILL.md for structure, style, and security, then merge. Elvis (@omarsar0) pointed me to an NVIDIA paper that tested whether those scans predict runtime quality.

In this sample, they barely agree with a separate LLM-judge score, and neither scan measures whether the skill helps an agent finish a task.

What the scans measure

On 145 real skills from NVIDIA’s internal catalogs and public registries:

• 94.5% pass the default structural C-grade (≥70 / 100).
• Only 48.9% reach 80.
• 86.2% pass the LLM-judge rubric at the same 70 cutoff.
• The two scores — deterministic structure vs LLM-judge quality — correlate at Spearman ρ = 0.14 (Pearson r = 0.08).

The correlation is the result that matters here. The scan methods disagree with each other. Skills with perfect frontmatter and vague instructions pass structure and fail the judge. Skills with rich content and missing metadata fail structure and pass the judge. Neither run tells you whether a live agent will discover the skill, call the right script, or finish the task.

I read the structural scan as analogous to compiler warnings: useful before execution, but not a behavioral test. The authors list runtime failure modes: the agent never loads the skill; it loads it and invokes the wrong script; it gets the right output and misreports it; it collides with a sibling skill; a model update silently changes how the agent reads the docs.

Skill Lift is a paired experiment

ACES (Agentic Continuous Evaluation of Skills) runs the same task twice under the same model, sandbox, workspace, and scorer: once with the target skill loaded, once without. Supporting skills and decoys stay fixed. Only the target is withheld in the baseline. The delta is Skill Lift.

Lift(metric) = mean_cases(score_with(metric) − score_without(metric))

Lift is first defined per metric. The paper’s composite is a separate aggregation of normalized metric scores; accuracy, goal accuracy, execution, behavior, and efficiency should also be inspected individually rather than treated as interchangeable units.

Across 947 paired cases from 58 of 64 production skills and four harnesses:

• Mean composite Skill Lift 0.2134 (95% CI [0.1967, 0.2301]).
• With-skill mean 0.7460 vs baseline 0.5326.
• Outcome-only lift (accuracy + goal accuracy) 0.1799.
• Positive in 689 / 947 cases (72.8%), zero in 171, negative in 87.

Negative lift is the regression signal: in 87 cases, the run with the skill scored worse than the baseline without it. A document scan cannot produce that comparison.

Process metrics explain the result

The paper also scores what happened during execution:

• Skill execution — did the agent read SKILL.md, run the expected script, in the right order, and recover from a failed tool call?
• Behavior check — free-form author assertions (“read the skill before executing”; “confirm before destructive ops”), judged yes/no on the trajectory.
• Skill efficiency — routing (only allowed skills) and tool waste (--help fishing, wrong-path ls, mid-task package installs).

Accuracy still moves (0.63 → 0.78). Goal accuracy moves more (0.47 → 0.69). But if you only score the final answer, you miss why a “working” skill is expensive or why a clean SKILL.md never fires.

NVIDIA’s product blog on the same methodology includes a useful cost comparison: one verified skill (jetson-optimize-memory) cut tokens 77% and wall time 54%; another (cuopt-install) increased tokens 120%. Skills are not free context.

Better models can shrink Skill Lift

The same-agent model slice is the result I keep coming back to: absolute scores can rise while marginal skill lift shrinks as the baseline model improves. A stronger agent already knows more of the procedure, so the skill’s added value falls even as the run looks better.

A dashboard that only shows the final score can credit the skill for a model upgrade. The paired delta separates the two.

Isolation, group runs, and routing

A skill in a solo workspace is not the same skill next to four neighbors. ACES splits:

• Isolation — only the target skill, which emphasizes the skill’s content.
• Group — target plus fixed decoys (api-debugger, log-triage, …), which also tests selection.

Lift_group − Lift_iso is the routing premium. A near-zero or negative result can indicate that the name and description do not discriminate. You can have a useful script and a skill the agent never picks. Progressive disclosure makes this worse at catalog scale: the agent only sees short descriptions until it decides to load. A scanner that checks description length (50–150 chars preferred) cannot check whether that description wins a lineup.

Harness results differ, but product differences are larger

Four primary harnesses, all positive, very different magnitudes (diagnostics, not a model ranking — each is a delta against its own baseline):

• OpenCode 0.36
• Claude Code 0.29
• Codex 0.13
• Terminus-2 0.09

The headline 0.2134 is case-weighted, not an average of those four. NVIDIA’s product blog also reports catalog-level results, but its point-scale summaries are not on the same scale as the paper’s normalized composite. They should be reported separately rather than used as a direct magnitude comparison.

To represent trajectories from different harnesses in one format, they normalize traces into ATIF (Agent Trajectory Interchange Format) — a shared step list of messages, tool calls, and observations. ATIF normalizes representation; it does not by itself make harness scores causally or statistically comparable. Each harness lift still has to be read against its own baseline, task mix, scorer, and execution environment.

What I would change in a skill repo

Treat evals/ the way you treat tests/:

1. Ship evals.json with the skill — prompts, ground truth, expected_behavior.
2. Cover explicit-name calls, implicit scenarios, noisy context, and a negative control where the skill should not fire.
3. Scan on every PR (cheap). Live paired runs when evals exist, on skill change, and after model upgrades.
4. Gate on Skill Lift, including the right to reject a skill that hurts.
5. Keep decoys in both arms so you are not measuring “a skill existed” instead of “this skill helped.”

The methodology is open in NVIDIA/SkillEvaluator (Tier 3 = the paired protocol). Verified NVIDIA skills and a living benchmarks.json sit in NVIDIA/skills.

If you maintain a skill library with SKILL.md files, evals, or an AGENTS.md vault, keep lint as a cheap first gate and add paired runs for behavioral evidence.