Richard Terry-Lloyd
A faded instruction document on a dark slate field, outlined in dim amber with a small clock in its corner and three dashed ghost copies stacked behind it standing in for the revisions that shipped after it was installed. A short dashed line runs from the document into a small terminal window marked with a shell prompt, and a longer dashed arrow carries on from the terminal to a large, brightly lit amber checkmark inside a glowing ring — a confident report of success produced from an out-of-date manual.
Back to blog

The Skill That Stopped Shipping

·Richard Terry-Lloyd
AIGovernanceSecurityOpen SourcePlatformEnterprise

A skill you installed six months ago is still telling your agent how the product worked six months ago.

The model moved on. The skill stayed where it was.

Two problems, and only one of them gets attention

Most people worry about the install. That is the right instinct. You want to know who wrote the thing and what it can reach once your team runs it. A skill from a source you cannot vouch for opens a hole in your enterprise, which is the conversation I had with Tanner Linsley in Salt Lake City and the reason provenance matters.

The other one catches you months later.

Your team installs a skill and leaves it there. The models keep changing underneath it. Whoever wrote it has moved on to something else. Nobody uninstalls anything, because nothing appears to be broken.

Your agent follows the old instructions and reports success.

That is the part that should bother you. Not a crash, not an error — a confident answer built on a manual that expired. A stale skill does not fail loudly. It just quietly makes your agent worse than it would have been with no skill at all, because now it is following instructions instead of asking.

You cannot read your way to the answer

Open the skill file and it looks fine. It is well written. It made sense when it was published.

Nothing in that file tells you whether it still describes reality.

So you cannot audit this by reading. You find it out the only way anyone finds out whether software works:

Run a real agent through a real scenario and score what comes back.

What we built for it

This is what AXIS is for — the Agent Experience Index Score. Sean Roberts drove it here at Netlify, and it does for agent experience roughly what Lighthouse did for page performance.

A scenario is a JSON file: a prompt, a rubric, and any setup the task needs. You run it against an agent and it scores what the agent could actually complete, split four ways so you learn why it struggled rather than just that it did:

  • Goal achievement (40%) — judged against your rubric
  • Environment (20%) — shell, filesystem, build tools
  • Service (20%) — your APIs, MCP tools, third-party dependencies
  • Agent (20%) — planning, tool selection, self-organization

Then you snapshot it as a baseline, and CI fails when the score regresses past a noise threshold. That is the whole point. A pass/fail tells you nothing on the day a model changes underneath you. A tracked number tells you the day it drops.

npm install -g @netlify/axis
axis init
axis run

Twenty-three coding agents are built in — Claude Code, Codex, Gemini, Cursor, Copilot and the rest — with a custom agent API for anything not on the list. It tests APIs, skills, MCP servers, and CLIs.

It is open source and free, developed in public at netlify/axis, with Auth0 among the first teams to run it and publish results. And it runs against any service, not only ours. We would rather the whole ecosystem had a number than have a private one.

None of this changes my advice

I still tell everybody to install the Netlify skill.

npx skills add netlify/context-and-tools --skill '*' --yes

We own the product and we keep that one current. The gap between what a model remembers from training and what a platform can do today is enormous, and a maintained skill is what closes it.

The skills worth worrying about are the ones whose author stopped updating them without telling you. There is no notification for that. There is no version bump, no deprecation warning, no email. The skill sits in your repo looking exactly as trustworthy as it did the day it went in.

Before your next agent project

Two questions, and they take an afternoon:

Who owns each skill this depends on, and when did they last ship a change to it?

If the answer to the first is "not sure" and the answer to the second is "a while ago," you already know what you are dealing with. Score it and find out how much it is costing you.

This is the same argument I keep making about putting guardrails in the platform rather than in a policy document. A rule nobody can measure is a rule nobody is following.