06: Test a FileMaker Skill: Evals and Baselines

'A case the baseline already passes isn't a test. It's a compliment to the model.' ~ my good-self, building a Skill to build Skills.

Your Skill works on your Mac. That's not the same as proving it helps. This lesson rebuilds it the way I build every Skill I release: evals first, a baseline, the validator, then three models.

In this post:

• Step 1: Install cadenceux-skill-creator

• Step 2: Answer 'Whose Skill Is It?'

• Step 3: Write the Evals Before the Skill

• Step 4: Validate the FileMaker Skill

• Step 5: Test Your FileMaker Skill on Three Models

• Skill Testing Vocabulary

Step 1: Install cadenceux-skill-creator

Back in June, Claude wrote me Skills that just wouldn't load. Hence a Skill to build Skills, recursive as that sounds. Install it the way lesson four showed, in the Code tab:

claude plugin marketplace add https://github.com/CadenceUX/cadenceux-skills.git
claude plugin install cadenceux-skill-creator@cadenceux

Start a new session in your Skill's folder, and open with the prompt from my session:

Let's create a Skill together using the cadenceux-skill-creator skill.
First ask me what the skill should do . . .

Step 2: Answer 'Whose Skill Is It?'

Before anything is named, it asks whose Skill it is. The answer decides the name, the author, the licence and how it's released. Tell it the Skill is your own.

It asks for your name and URL, a name prefix (none is fine) and a licence: CC BY 4.0 for a Skill that's mostly writing, MIT for one that's mostly scripts. CadenceUX's prefixes and attribution stay out. Your name is on it.

Example: The eval for this has a fictional Developer, Sam Lee, asking for a Skill that explains a team's FileMaker naming conventions. Haiku 5.5, Sonnet 5.5 and Opus 5.5 all asked whose Skill it was, named it filemaker-naming-conventions with no CadenceUX prefix, and put Sam's name on it. Haiku and Opus also raised the overlap with ADT's filemaker-standards Skill.

That overlap is lesson five in action. ADT is the foundation, and your Skill is your layer on top.

Step 3: Write the Evals Before the Skill

This is the step most Skills skip. Before you change a line, Claude runs three or more of your real prompts with no Skill loaded. That's the baseline. Each failure becomes a case in evals/evals.json: what a right answer must say, and what it must not. Then a few near-misses: prompts that sound close but shouldn't trigger your Skill.

One case from cadenceux-skill-creator's own evals, trimmed:

{
  "id": "frontmatter-new-skill",
  "prompt": "Write the frontmatter for a new CadenceUX skill called claude-filemaker-helper that explains FileMaker error codes.",
  "must": ["rejects or renames the name because it contains 'claude'"],
  "must_not": ["keeps 'claude' in the name"]
}

Example: That must_not is real. With no Skill loaded, the baseline kept the reserved word 'claude' in the name. Another case passed with no Skill at all, so it wasn't testing anything the Skill adds.

Step 4: Validate the FileMaker Skill

Claude runs the validator from cadenceux-skill-creator's folder against yours:

python3 scripts/validate_skill.py --own /path/to/your-skill

It checks the limits from lesson one, the body length, how deep the references go, and that the version agrees everywhere it's written. It prints the fix for each failure. --own drops the CadenceUX-only checks and warns if CadenceUX branding has leaked into your Skill. Fix, re-run, and carry on only when it's clean.

Step 5: Test Your FileMaker Skill on Three Models

Each eval case runs in a fresh subagent: once with no Skill, and once on each model with it. Haiku shows whether the Skill gives enough guidance. Opus shows whether it over-explains. A Skill that doesn't beat its baseline isn't earning its place.

When a case fails, fix the Skill, not the prompt, and re-run. Then tune the description against 8–10 prompts that should trigger it and 8–10 that shouldn't.

Example: When I built the claris-filemaker-pro Skill, the baseline passed 2 of 8 cases. With the Skill, Sonnet 5.5 and Opus 5.5 passed all 10, and Haiku 5.5 climbed from 6 to 9 to 10. A smaller model obeys the checklist it copies at the start, not the rule buried in a reference file. Put the rule that matters at the top.

The evals stay with the Skill. Mine are run again on every release.

Skill Testing Vocabulary

Eval. A test case: a real prompt, with what the answer must and must not contain.

Baseline. The same evals run with no Skill loaded, to prove the Skill adds something.

Near-miss. A prompt that sounds close but shouldn't trigger the Skill.

Subagent. A fresh Claude that runs one eval with no memory of the build.

Validator. The script that checks a Skill against the upload limits and its own version.

Takeaway

A compliment to the model isn't a test. Write down what the baseline got wrong, make the smallest Skill that fixes it, validate it, and prove it on three models before your name goes on it.

A tested Skill is the one worth sharing. Continue with my next post, Your First FileMaker Skill on GitHub.

More to come . . .