Skip to main content

Create Test Skills

since 1.4.0

Waterwheel tests are prose, not scripts — so the way to reuse work is not a helper function or a page object. It is a test skill: a Markdown file holding a named piece of instruction that the agent pulls in when a test needs it.

Skills are how Waterwheel avoids test scripts entirely. Everything you would normally put in shared code — a login helper, a workaround for a widget the browser automation reads oddly — stays prose, stays readable by anyone on the team, and lives in exactly one file.

Before you start

You need a running, configured container — Quick Start steps 1 to 3 — and you should have written and run a test of your own, as in Write Your First Test. This guide does not re-explain uploading tasks, domain permissions, or reading results.

What Skills Are For​

A skill earns its place by doing one of two jobs.

JobThe problemWhat a skill does
Bridge an LLM gapA particular interaction is one the agent gets wrong or does inconsistently — an ambiguous control, a widget whose accessible structure is misleading, an outcome that has to be confirmed a specific waySpell out the exact procedure once, in enough detail that the agent stops improvising
Remove repetitionTen tests all begin by logging in, and the login steps are copy-pasted into all ten task filesDescribe the flow once as a skill; each test refers to it by name

Both are the same mechanism. The difference is what you are writing: the first kind is a precise procedure, the second is an abstracted process.

Two different things are called "skills"

This page is about test skills, which live inside the agent container and instruct the test agent. They are unrelated to the Waterwheel skills you install into a code agent such as Claude Code, which drive the container from the outside.

How the Agent Uses a Skill​

The agent always knows a skill's name and its one-line description — nothing more. When it reads a test whose work matches a description, it asks for that skill by name and receives the full instructions before acting.

Two consequences worth understanding before you write one:

  • The description is the trigger. It is the only part of your skill the agent sees while deciding whether the skill applies. Write it as guidance to the agent ("Use when a task asks to…"), not as documentation for a human.
  • A long skill body costs nothing until it is used. A test that never touches the clipboard never pays for the clipboard instructions. This is why you can afford to be genuinely thorough in a skill body in a way you could not in a task file.

The Two Tiers​

TierWho owns itNamedPurpose
Built-inShips inside the image, versioned with the agentww:<name>Teaches the agent to work around limitations of the underlying browser automation
UserYou<name>, exactly as you write itYour application's flows and your own LLM-gap fixes

The ww: prefix is reserved. It is what lets you name a skill anything you like without ever colliding with a shipped one — a user skill called form-and-dialog and the built-in ww:form-and-dialog coexist as two separate skills, and neither hides the other. A user skill whose own name starts with ww: is refused and skipped, so do not use that prefix in your own files.

The built-in skills​

Three ship with 1.4.0. Each covers an interaction where the raw browser tooling is not enough on its own:

SkillCovers
ww:clipboard-captureVerifying what a copy button actually copied, rather than what the page happens to display
ww:form-and-dialogFilling forms and confirming that a button click really did something — a dialog closed, a row appeared, a URL changed
ww:take-browser-screenshotCapturing a screenshot and filing it under the current test's own folder

List them at any time:

docker exec -it waterwheel-agent display-test-skills
clipboard-capture (built-in)
form-and-dialog (built-in)
take-browser-screenshot (built-in)

Read one before you write your own — they are the best available examples of the level of detail a skill body wants:

docker exec -it waterwheel-agent display-test-skills --show form-and-dialog

Anatomy of a SKILL.md​

A skill is a folder containing a single SKILL.md: YAML front matter, then the body.

SKILL.md
---
name: login-test-site
description: Log a named user into the test site. Use whenever a task requires an authenticated session.
---

# Login to Test Site
<the full instructions>
FieldRequiredNotes
nameYesKebab-case identifier. This is the name tests and the agent use.
descriptionYesOne line, agent-facing. Say what the skill does and when to use it.

Everything after the front matter is the body: ordinary Markdown, and the only part the agent reads once it decides the skill applies. There is no step syntax and no assertion API — the same prose rules as a task file, with more room to be exact.

Keep the folder name and the name field identical

The agent identifies a skill by its front-matter name; the CLI commands identify it by its folder name. Nothing breaks if the two differ, but you end up deleting login-flow to remove a skill the agent calls login-test-site. Use one spelling for both.

Example 1 — Remove Repetition​

Login is the classic case: every authenticated test needs it, and nobody wants it pasted into twenty files. Write it once, parameterized by the user it should log in as.

SKILL.md
---
name: login-test-site
description: >
  Log a given user into the test site. Triggered by phrases like
  "log into test site with user X", "sign in test user X", or
  "login to the test site as X". Verifies the user exists in context,
  opens the login page, submits the user's credentials, and confirms
  the session lands on the test site home page.
---

# Login to Test Site

Reusable test skill that authenticates a **given user** against the test site. The
"given user" is whatever name follows the trigger phrase (e.g. `testuser`,
`admin_user`); throughout the steps below, substitute that name for `<user>` and
read its fields from `${context.<user>}`.

## Steps

1. **Get Test User:** Verify that the given user exists in the context manager — for
   example, if the given user is `testuser`, the variable `${context.testuser}` must
   exist. If it is missing, fail the test immediately with the reason:
   "Test user credential isn't found in context."
2. **Open Login Page:** Open a browser tab and navigate to `LOGIN_URL`. Fail with
   reason: "Failed to open the login page." if the page does not load.
3. **Login:** Fill the `username` field with `${context.<user>.username}` and the
   `password` field with `${context.<user>.password}`, then submit the login form.
   Fail with reason: "Failed to submit login credentials for the given user." if the
   form cannot be submitted.
4. **Verify Logged In:** Verify the browser URL becomes `TEST_URL`. Fail with reason:
   "Login did not redirect to the test site." if the URL does not become `TEST_URL`.
   Otherwise, complete the skill as SUCCESS.

Four things make this reusable rather than merely extracted:

  • It takes an argument. <user> is a placeholder the calling test fills in, so one skill serves testuser, admin_user, and whatever account you add next.
  • It reads its inputs from context, not from itself. No credential is written in the file. LOGIN_URL and TEST_URL are global constants and the accounts are preset context values — so the same skill authenticates against dev and staging unchanged.
  • It validates before it acts. Step 1 fails loudly on a missing account instead of submitting an empty form and reporting a confusing login failure.
  • Every step carries its own failure reason. When a run breaks inside a shared skill, the result string tells you which stage of it broke.

Load it into the agent. The content is piped from your host, so use -i without -t:

cat ./SKILL.md | \
  docker exec -i waterwheel-agent load-test-skills -name login-test-site
Saved content to: /agent/skills/login-test-site/SKILL.md

Confirm it landed:

docker exec -it waterwheel-agent display-test-skills
clipboard-capture (built-in)
form-and-dialog (built-in)
take-browser-screenshot (built-in)
login-test-site

Now a test that needs an authenticated session is two lines instead of five:

view-profile.md
---
name: View Profile
id: 1
---

# View Profile

1. Load the `login-test-site` skill and log into the test site with user `testuser`.
2. Verify the account menu shows "testuser".
3. Open the profile page and verify the email matches `${context.testuser.username}`.

Example 2 — Bridge an LLM Gap​

The second kind of skill exists because a specific interaction goes wrong without one. Clicking a link by its visible text sounds trivial, but the text you see is often a child node inside the link, several links on the page start with similar words, and the agent may re-derive its target between finding it and clicking it. The fix is a procedure precise enough to leave no room for improvisation:

SKILL.md
---
name: oa-link-click
description: Click a link identified by its visible text. Use when a task step says to click a link `<linktext>`, or click on a link/hyperlink whose label or displayed text is given. Resolves the link's ref from the ARIA tree even when the matched text is a child text node of the link, and clicks it.
---

### Clicking a link by its text

Use this when a step asks to click a link identified by its visible text `<linktext>`
(e.g. "Click the `<linktext>` link"). Follow these steps exactly.

1. **Snapshot.** Call `take_verification_snapshot` to get the current ARIA tree and its
   `snapshot_id`. Use only the token after `ref=` as the click target. Keep this snapshot
   open until after the click.

2. **Find the link line in the snapshot.** Scan for the single `link`-role line whose
   accessible name **begins with** `<linktext>`.
   - If `<linktext>` is on a child node (`text`, `generic`, `strong`), use the enclosing
     `link` line, not the child's ref.
   - If no name begins with `<linktext>`: screenshot and **fail** with
     "No link found for text `<linktext>`."
   - If two or more qualify and you cannot tell which is intended: screenshot and **fail**
     with "Ambiguous link text `<linktext>` — multiple matches."

3. **Confirm and record.** On that same line, read its `ref` and the `/url` beneath it. If a
   destination is expected and the `/url` contradicts it, screenshot and **fail** with
   "Resolved link `<linktext>` points to the wrong destination." Re-read the snapshot line
   rather than working from memory.

4. **Click immediately.** As the very next action, with no other tool call in between, click
   using the exact `ref` token from the line you just recorded.

5. **Release the snapshot.** Call `complete_verification` with the `snapshot_id` and
   `purpose: "snapshot_release"`.

What makes this shape of skill work:

  • It names the failure modes and what to do about them. "No match" and "ambiguous match" each get an explicit outcome, so the agent fails with a useful reason instead of clicking its best guess.
  • It constrains ordering, not just actions. As the very next action, with no other tool call in between is the whole point of step 4 — it closes the gap where a stale target could be substituted.
  • It is emphatic where the agent tends to drift. Bold and "exactly" are not decoration here; they mark the instructions that stop being followed first.
  • It ends by cleaning up. Releasing the snapshot keeps the run's token cost flat. See Tuning.

Load it the same way, then any test can say "Click the Order history link" and get the same careful behavior every time — that one skill fixes the interaction across your whole suite.

Getting the Agent to Actually Use a Skill​

The agent decides for itself whether a skill applies, which is exactly right for a judgment call ("this task happens to involve the clipboard") and weaker than you want for a requirement ("always authenticate this way").

Two things make it reliable:

  1. Write the description as a trigger. Include the phrasings your tests use. The login-test-site description above lists "log into test site with user X", "sign in test user X", and "login to the test site as X" on purpose.
  2. Name the skill in the test when it is mandatory. Load the "login-test-site" skill and log in as testuser leaves nothing to inference. Use this for any skill that must run — it is the difference between a hint and an instruction.
Check the skill was used, not just that the test passed

The result string in test-results.json describes what the agent did. If a test that should have gone through your login skill reports its own improvised login steps, the skill did not trigger — tighten the description, or name the skill in the task. get-failure-detail shows every step and tool call for a failed test, including the skill load.

Swap Skills Per Environment​

The agent reads user skills from /agent/skills by default. Point SKILLS_DIR somewhere else and the same tests resolve to a different implementation:

docker run -d --name waterwheel-agent \
  -e AI_API_KEY \
  -e SKILLS_DIR=/agent/skills-staging \
  taojdcn/duotail-waterwheel:1.4.0

Your task files still say login-test-site; which login they get depends on the directory. This is the reason skills are worth extracting even when a flow appears in only one test — the flow becomes something you can vary per environment without editing a single test.

Built-in skills are never affected by SKILLS_DIR. Repointing it — or mounting your own folder over /agent/skills — leaves ww:clipboard-capture, ww:form-and-dialog, and ww:take-browser-screenshot in place.

Managing Skills​

CommandDoes
load-test-skillsCreate a skill from piped content
display-test-skillsList every skill, or print one's SKILL.md
delete-test-skillsRemove your skills by name, or all of them
delete-builtin-skillsRemove a shipped skill by exact name

Updating a skill needs --force, since an existing folder is left alone by default:

cat ./SKILL.md | docker exec -i waterwheel-agent \
  load-test-skills -name login-test-site
Skipping existing skill folder: /agent/skills/login-test-site (use --force to overwrite)
cat ./SKILL.md | docker exec -i waterwheel-agent \
  load-test-skills -name login-test-site --force

Removing one:

docker exec -it waterwheel-agent delete-test-skills -n login-test-site
Deleted skill: login-test-site
Done. Deleted 1 skill(s), skipped 0.

-n takes a comma-separated list; -a clears every user skill and touches no built-in. A name that matches nothing is reported and skipped rather than treated as an error.

delete-builtin-skills removes a capability from the image

Deleting a built-in is occasionally the right call — you have written a replacement and want only one of the two active. But the shipped skills exist because the agent handles those interactions poorly without them, and the only way back is recreating the container. There is deliberately no -a for built-ins; each must be named exactly.

Gotchas​

SymptomCause
Skill file arrives truncated or mangled-it was used while piping. Every command reading stdin needs plain -i.
Skill never triggersThe description does not read as a trigger for the wording your tests use — or the skill is a requirement and the task should name it outright.
Skill is missing after loadingThe front-matter name starts with ww:, which is reserved; such a skill is skipped.
Deleting by name reports nothing to deleteThe CLI matches the folder name, which may differ from the front-matter name. display-test-skills shows folder names.
A skill body's instructions are being ignored mid-runThe body is long and unemphatic. Put the load-bearing constraints early and mark them.

Token Efficiency on Anthropic Models​

Everything above says a skill body costs nothing until it is used. That holds — but on Anthropic models, used costs more than it first appears, and the difference is worth understanding before you build a long suite around large skills.

Anthropic caches only a prefix of each request, up to an explicit cache breakpoint. Waterwheel places that breakpoint over the material that is fixed for the whole run — the system prompt, the instruction files, the tool definitions — and it is settled before the first step runs. A skill loaded partway through a test arrives after that boundary, so its body never enters the cache. It is billed as ordinary input tokens on every API call from the load onwards, until the test finishes or context compression discards it.

The cost is therefore not the skill body once — it is the skill body multiplied by the number of steps that follow the load, and it grows linearly with the length of the scenario. A 2,000-token skill pulled in at step 3 of a 40-step test is paid for roughly 37 times.

This only matters for long scenarios and substantial skill bodies; a short skill in a ten-step test is noise. When it does matter, there are three levers:

LeverUse it when
Run in efficiency modeAlways. CONTEXT_COMPRESSION=true caps how many calls carry the skill text forward — Claude's recommended thresholds are deliberately aggressive for this reason. See the Provider Configuration Guide.
Move the instructions to extra-instructions.mdSeveral tests need the same prose. That file is appended to the system prompt, which sits inside the cached prefix, so its tokens are cached for the whole run instead of re-billed per call. See Configuration Files.
Write the instructions into the task fileExactly one test needs them. The prose is in place before the run starts rather than injected mid-test, and you also save the extra round trip the agent spends asking for the skill.
This is an Anthropic-specific trade-off

The pattern above follows from how Anthropic's prefix caching works. It is not a reason to avoid skills — the two jobs they do are still worth it, and a skill that fixes an interaction the agent otherwise gets wrong saves far more than it costs in retried steps. It is a reason to keep frequently-loaded, cross-suite prose in extra-instructions.md and reserve skills for procedures that genuinely belong to a subset of tests.

Next Steps​