Command Reference

This page documents every command in the container, listed alphabetically. If you have not run a test of your own yet, start with Write Your First Test — it walks through the commands you need in the order you need them.

Commands

The container provides the following commands to help you run web tests.

Command	Usage
`check-test-result`	Check test results
`config-agent`	Configure AI provider, browser permissions, and access to a test website deployed on the host machine
`config-ai-provider`	Apply an AI provider and model without interactive prompts
`delete-builtin-skills`	Delete built-in skill folders by name
`delete-test-skills`	Delete user-loaded skills by name or all
`display-ai-config`	Show the effective AI provider, model, and token mode
`display-test-skills`	List installed skills or print a skill's `SKILL.md`
`enable-test-on-host`	Enable testing against a web app on the host machine
`get-failure-detail`	Get failure details
`load-test-skills`	Load a skill into the agent as a `SKILL.md`
`manage-global-constants`	Manage global constants
`manage-test-files`	Manage markdown test task files
`output-context-variables`	Output user-scoped context values from the latest run
`preset-context`	Manage preset context variables and test flow
`reset-test-config`	Delete test tasks and/or instruction files to reset state
`run-qa`	Run QA tasks
`set-domain-permission`	Set the browser domain allowlist from a domain list
`stop-qa`	Stop current QA run
`upload-instruction-file`	Upload an instruction or config file into the agent
`upload-test-task`	Upload a markdown test task into the agent

Running the Commands

Every command below runs inside the container. From your host machine, prepend docker exec -it waterwheel-agent:

Execute run-qa on host machine
docker exec -it waterwheel-agent run-qa

Each command's synopsis is shown in its in-container form (the signature), while the Examples are written as ready-to-paste docker exec commands.

Which docker exec flags to use

Situation	Flags	Example
Normal command	`-it`	`docker exec -it waterwheel-agent run-qa`
Piping content from the host into the container	`-i` only	`cat ./login.md \| docker exec -i waterwheel-agent upload-test-task login.md`
Passing an environment variable	`-e` before the container name	`docker exec -e SKILLS_DIR=/tmp/skills -it waterwheel-agent display-test-skills`

Adding -t while piping from the host allocates a TTY, which corrupts the piped input. Use plain -i for the upload-* and load-test-skills commands.

Silence Docker CLI hint messages

Docker prints occasional hint messages above your command's output, which is noisy when you are reading test results or piping output into another tool. Set DOCKER_CLI_HINTS to false in the shell you run these commands from:

macOS / Linux
Windows (PowerShell)
Windows (CMD)

export DOCKER_CLI_HINTS=false

$env:DOCKER_CLI_HINTS = "false"

set DOCKER_CLI_HINTS=false

This lasts only for the current terminal session. To make it permanent, add the line to your shell profile (~/.zshrc, ~/.bashrc, or your PowerShell profile).

Using Docker Compose

If you brought the agent up with Docker Compose, replace docker exec ... waterwheel-agent with docker compose exec ... waterwheel-agent in every example. The command and its arguments are otherwise identical.

`check-test-result` Command

since 1.1.0

check-test-result prints the overall outcome of the latest run, or reports the current run status if a test is still in progress. It is a helper command for code agent integration.

check-test-result

Output

Condition	Output
`run-qa` is currently active	A message indicating testing is in progress, including the orchestrator PID
Test results exist	The exit condition summarizing the run outcome
`test-results.json` missing, `agent.log` missing	`ℹ️ No test results found.`
`test-results.json` missing, `agent.log` exists	`ℹ️ No test results found.` followed by the full content of `agent.log`

Example

docker exec -it waterwheel-agent check-test-result

`config-agent` Command

since 1.1.0

The config-agent command allows users to configure the AI provider, browser permissions, and access to a test website deployed on the host machine.

config-agent

config-agent is interactive, so the -t flag is required:

docker exec -it waterwheel-agent config-agent

First Run (no mode configured)

If no AI provider mode has been set yet, the script skips the main menu and goes directly into AI provider mode selection. The user must choose and confirm a mode before the main menu appears.

Returning Runs

========================================
  Waterwheel Agent Configuration
========================================

  Select an item to configure:

  1. AI provider mode [Anthropic Default Mode]
  2. Config domain permissions
  3. Test web app on host [disabled]
  0. Exit

----------------------------------------
  Choice:

The current state of option 1 (active mode name) and option 3 (enabled/disabled) is shown inline. Enter the number and press Enter.

Option 1 — AI Provider Mode

Selects the AI provider and model the agent uses. The AI provider must match the provider of your AI API key. When no provider mode is configured, the script enters initial mode automatically. Once a provider mode is set, the menu only shows modes belonging to the same provider as the current mode. The Manual customized mode is not shown.

Option 2 — Config Domain Permissions

This option allows users to configure the agent's domain access permissions. If the agent tries to access any URLs not whitelisted through domain permissions, the test fails immediately.

See the URL Permissions section for full details on configuring allowed domains.

Option 3 — Test Web App on Host

If the website under test is hosted on your local machine, for example at http://localhost:8080, enable this option so the agent in the Docker container can interact with it.

`config-ai-provider` Command

since 1.3.0

config-ai-provider applies an AI provider and model without the interactive prompts of config-agent. It is the scripting-friendly way to set up the AI provider, for example in CI or a one-off setup command.

config-ai-provider --provider <provider> --model <model> --mode <default|efficiency>

Options

Option	Description
`--provider <value>`	AI provider to use (for example, `anthropic` or `openai`). Must match the provider of your AI API key
`--model <value>`	Model name to use
`--mode <value>`	`default` for the standard prompt mode, or `efficiency` for the token-efficiency mode

Behavior

The provider must match the provider of your AI API key.
Provider locking applies: once a provider is configured, switching to a different provider requires a new container.
For provider-specific extras (such as a custom base URL or a localized system prompt), use the interactive config-agent command instead.

Examples

# Apply Anthropic default mode with an explicit model
docker exec -it waterwheel-agent \
  config-ai-provider --provider anthropic --model claude-sonnet-4-6 --mode default

# Apply OpenAI token-efficiency mode
docker exec -it waterwheel-agent \
  config-ai-provider --provider openai --model gpt-5.4 --mode efficiency

`delete-builtin-skills` Command

since 1.4.0

delete-builtin-skills removes built-in skill folders that ship with the image, under $AGENT_PATH/builtin-skills. It deletes a comma-delimited list of skill names (-n), removing each exactly-matched folder. Unlike delete-test-skills, there is no -a/--all option — built-in skills can only be removed by exact name. A leading ww: prefix on a name is ignored, so ww:foo targets the folder foo.

delete-builtin-skills -n <name1,name2,...>

Options

Option	Description
`-n`, `--names <names>`	Comma-delimited built-in skill folder names to delete
`-h`, `--help`, `h`, `help`	Show usage help

Behavior

-n is required; there is no bulk-delete option.
Built-in skills live at $AGENT_PATH/builtin-skills (this location is not configurable via $SKILLS_DIR).
A leading ww: prefix is stripped from each name before matching, so ww:foo and foo are equivalent.
Each name is trimmed of surrounding whitespace, so -n 'a, b' works as expected.
A matching folder <builtin-skills>/<name> is removed recursively; a name with no matching folder is reported and skipped (the command still exits 0).
Any name containing a path separator (or ./.., evaluated after the ww: prefix is stripped) is rejected and the command exits non-zero before deleting anything further.
On completion it prints a summary of how many skills were deleted and skipped.

Examples

# Delete a single built-in skill
docker exec -it waterwheel-agent delete-builtin-skills -n login-flow

# Delete several built-in skills at once
docker exec -it waterwheel-agent delete-builtin-skills -n login-flow,checkout-flow

# A "ww:" prefix targets the unprefixed skill name
docker exec -it waterwheel-agent delete-builtin-skills -n ww:login-flow

`delete-test-skills` Command

since 1.4.0

delete-test-skills removes skill folders that were loaded via load-test-skills. It either deletes a comma-delimited list of skill names (-n) or clears every user-defined skill (-a), removing each matching folder under the skills directory. The skills directory is $SKILLS_DIR when set, otherwise $AGENT_PATH/skills. Only user-loaded skills are affected — built-in skills under $AGENT_PATH/builtin-skills are never touched.

delete-test-skills -n <name1,name2,...>
delete-test-skills -a

Options

Option	Description
`-n`, `--names <names>`	Comma-delimited skill folder names to delete
`-a`, `--all`	Delete all user-defined skills under the skills directory (mutually exclusive with `-n`)
`-h`, `--help`, `h`, `help`	Show usage help

Behavior

Exactly one of -n or -a is required; supplying both exits non-zero.
With -a, every immediate subfolder of the skills directory is deleted; if the directory is missing or empty, the command still exits 0.
Each name is trimmed of surrounding whitespace, so -names 'a, b' works as expected.
A matching folder <skills-dir>/<name> is removed recursively; a name with no matching folder is reported and skipped (the command still exits 0).
$SKILLS_DIR overrides the default $AGENT_PATH/skills location.
Any name containing a path separator (or ./..) is rejected and the command exits non-zero before deleting anything further.
On completion it prints a summary of how many skills were deleted and skipped.

Examples

# Delete a single skill
docker exec -it waterwheel-agent delete-test-skills -n login-flow

# Delete several skills at once
docker exec -it waterwheel-agent delete-test-skills -n login-flow,checkout-flow

# Delete all user-defined skills
docker exec -it waterwheel-agent delete-test-skills -a

# Use a custom skills directory (pass the variable with docker exec -e)
docker exec -e SKILLS_DIR=/tmp/skills -it waterwheel-agent \
  delete-test-skills -n login-flow

`display-ai-config` Command

since 1.3.0

display-ai-config prints the effective AI runtime settings as JSON. Use it to confirm which provider, model, and token mode the agent will use before running tests.

display-ai-config

Output

Returns a JSON object with exactly these keys:

Key	Description
`aiProvider`	The active AI provider
`aiModel`	The active model
`tokenMode`	`default` for the standard prompt mode, or `efficiency` for the token-efficiency mode. Empty when not configured

Example

docker exec -it waterwheel-agent display-ai-config

`display-test-skills` Command

since 1.4.0

display-test-skills lists installed skills, or prints a single skill's SKILL.md. Skills are read from two locations: $AGENT_PATH/builtin-skills (shipped with the image) and the user skills directory — $SKILLS_DIR when set, otherwise $AGENT_PATH/skills (loaded via load-test-skills). Without arguments it lists every skill in both, tagging built-in skills with (built-in). With --show <skill-name> it prints that skill's SKILL.md, preferring a user skill over a built-in one of the same name, and prints No skill is matched. when neither exists.

display-test-skills [--show|-s <skill-name>]

Options

Option	Description
`-s`, `--show <skill-name>`	Print the named skill's `SKILL.md` instead of listing
`-h`, `--help`, `h`, `help`	Show usage help

Behavior

List mode enumerates skill folders under builtin-skills/ (marked (built-in)) then the user skills directory; if neither has any, it prints No skills installed..
Show mode looks up <skills-dir>/<name>/SKILL.md first, then builtin-skills/<name>/SKILL.md, so a user skill shadows a built-in of the same name.
$SKILLS_DIR overrides the default $AGENT_PATH/skills location for the user skills directory.
A skill name containing a path separator (or ./..) is rejected.
When the named skill is not found in either location, it prints No skill is matched. and exits 0.

Examples

# List all installed skills (built-in and user)
docker exec -it waterwheel-agent display-test-skills

# Print a specific skill's SKILL.md
docker exec -it waterwheel-agent display-test-skills --show login-flow

# Short flag form of --show
docker exec -it waterwheel-agent display-test-skills -s login-flow

`enable-test-on-host` Command

since 1.3.0

enable-test-on-host enables testing against a web app deployed on your host machine, mirroring the Test web app on host action in config-agent. Use it when the website under test is hosted on your local machine, for example at http://localhost:8080, so the agent inside the Docker container can reach it.

When enabled, the command:

Instructs the agent to rewrite localhost to host.docker.internal during test runs.
Rewrites localhost to host.docker.internal in your domain allowlist.
Rewrites every localhost value in your global constants to host.docker.internal.

enable-test-on-host

Behavior

Re-running is idempotent: host testing is not enabled twice if it is already enabled.
The domain allowlist and global constants are only rewritten if they exist and contain localhost.

Example

docker exec -it waterwheel-agent enable-test-on-host

`get-failure-detail` Command

since 1.1.0

When a test failure is reported by a previous run, this command prints a full diagnostic report for the first failed test found in $AGENT_PATH/outputs/test-results.json. If a test run is still in progress, it reports that instead. It is another helper command for code agent integration.

get-failure-detail [-d]

Options

Option	Description
`-d`	Include API log (`/agent/outputs/api-log.json`) at the end of the report if file exists

Output when a failed test is found

Each section is printed in order. Missing files are reported inline and do not abort the output.

Section	Source
Failed Test Summary	The failed test JSON object from `test-results.json`
Test Detail	`/agent/tasks/<test-file>`
Test Steps	`/agent/outputs/<test-file-stem>_log.json`
Test Context	`/agent/outputs/test-context.json`
Agent Log	`/agent/outputs/agent.log`
API Log (only with `-d`, only if file exists)	`/agent/outputs/api-log.json`

Output when no failure

Condition	Output
`run-qa` is currently active	A message indicating testing is in progress, including the orchestrator PID
`test-results.json` missing, `agent.log` missing	`ℹ️ No test results found.`
`test-results.json` missing, `agent.log` exists	`ℹ️ No test results found.` followed by the full content of `agent.log`
Run did not complete	`⚠️ Run did not complete:` followed by the exit condition
Test results exist, no failed tests	`✅ No failed tests found in test results.`

Examples

# Print the diagnostic report for the first failed test
docker exec -it waterwheel-agent get-failure-detail

# Include the API log in the report
docker exec -it waterwheel-agent get-failure-detail -d

`load-test-skills` Command

since 1.4.0

load-test-skills creates a skill folder named <skill-name> under the agent's skills directory and writes content read from stdin into a SKILL.md inside it. This lets you push a skill into the agent without editing files in place. An existing skill is left untouched unless you pass --force. For what belongs in a SKILL.md, see Create Test Skills.

load-test-skills -name <skill-name> [--force]

Behavior

Content is read from stdin and written to the skill's SKILL.md.
A skill name containing a path separator (or ./..) is rejected.
An existing skill folder is skipped unless --force (-f) is given.
Missing parent directories are created automatically.

Examples

Content is piped in from the host, so use docker exec -i without -t:

# Create a skill from a file on the host
cat ./SKILL.md | docker exec -i waterwheel-agent load-test-skills -name login-flow

# Overwrite an existing skill
cat ./SKILL.md | docker exec -i waterwheel-agent load-test-skills -name login-flow --force

# Write a skill inline without a local file
printf '# Login flow\n\nSteps...\n' | \
  docker exec -i waterwheel-agent load-test-skills -name login-flow

`manage-global-constants` Command

since 1.1.0

The manage-global-constants command provides a simple entry point to configure global context environment variables used by tests.

manage-global-constants <operation> [args]

Operations

Operation	Arguments	Description
`list`	—	Display all current values, or a message if none are set
`set`	`KEY=value,...`	Set one or more key/value pairs (comma-delimited)
`delete`	`KEY,...`	Delete one or more keys by name (comma-delimited)
`clear`	—	Delete the entire context file
`help` / `h`	—	Show usage

Examples

# List all values
docker exec -it waterwheel-agent manage-global-constants list

# Set multiple values (quoted and unquoted)
docker exec -it waterwheel-agent manage-global-constants \
  set BASE_URL="https://staging.example.com",TENANT=acme,SUPPORT_EMAIL=qa@example.com

# Set nested values with dotted keys
docker exec -it waterwheel-agent manage-global-constants \
  set user.username=qa_user,user.password=secret

# Overwrite an existing key
docker exec -it waterwheel-agent manage-global-constants set TENANT=newcorp

# Delete specific keys (unknown keys produce a warning, known keys are still deleted)
docker exec -it waterwheel-agent manage-global-constants delete BASE_URL,TENANT

# Remove all values (also deletes the file)
docker exec -it waterwheel-agent manage-global-constants clear

Operation Rules

Key names are case-sensitive. We recommend using uppercase names.
Values may be quoted or unquoted.
Dotted keys create nested objects (for example, user.username=qa_user becomes { "user": { "username": "qa_user" } }).
Unknown key names in delete print a warning but do not cause an error; any found keys are still deleted.

`manage-test-files` Command

since 1.2.0

The manage-test-files command manages the markdown test task files the agent runs. Use it to add, list, and remove .md task files without copying files into the container by hand.

manage-test-files <operation> [args]

Operations

Operation	Arguments	Description
`list`	—	List `.md` task files with 1-based indexes
`add`	`path1,path2,...`	Add markdown files from file paths and direct directory children; non-markdown files are ignored
`delete`	`selector1,selector2,...`	Delete by 1-based index or exact filename (basename only), best effort
`clear`	—	Delete all markdown test files
`help` / `h`	—	Show usage

Examples

# List current markdown test files
docker exec -it waterwheel-agent manage-test-files list

# Delete by index
docker exec -it waterwheel-agent manage-test-files delete 1,3

# Delete by filename and index in one call
docker exec -it waterwheel-agent manage-test-files delete login-flow.md,2

# Clear all markdown test files
docker exec -it waterwheel-agent manage-test-files clear

The add operation resolves its paths inside the container, so it is only useful when the files are already there — for example under a volume you mounted with Docker Compose:

# Add two markdown files that exist inside the container
docker exec -it waterwheel-agent \
  manage-test-files add /agent/import/test-a.md,/agent/import/test-b.md

# Add all direct markdown files from a container directory (subdirectories are ignored)
docker exec -it waterwheel-agent manage-test-files add /agent/import

Adding a task from the host

If you started the container without volume mounts, add cannot see your host files. Pipe the task in with upload-test-task instead:

cat ./test-a.md | docker exec -i waterwheel-agent upload-test-task test-a.md

Operation Rules

Only .md files are listed and managed.
add overwrites an existing file with the same destination name silently.
Directory import includes only direct child files, not subdirectories.
delete is best effort: invalid selectors print warnings while valid selectors are still deleted.
Source paths passed to add are resolved inside the container, not on the host machine. Make sure the files are reachable inside the container (for example, under a mounted volume) before referencing them, or use upload-test-task to push a file in from the host.

`output-context-variables` Command

since 1.3.0

output-context-variables prints context values produced by the latest run as a flat JSON object. Tests store values such as AI-generated usernames and passwords in /agent/outputs/test-context.json; this command extracts those entries so you can reuse them outside the container (for example, to log in with a freshly registered account).

output-context-variables

Options

Option	Description
`-h`, `--help`, `h`, `help`	Show usage help

Errors

Condition	Result
`run-qa` is currently active	Prints an error that exporting context variables while testing is in progress isn't supported, and exits non-zero
`test-context.json` is missing	Prints an error that no `test-context.json` was found, and exits non-zero

Example

docker exec -it waterwheel-agent output-context-variables

It prints:

{
  "testuser": {
    "username": "test.user21",
    "password": "123456789Ab#",
    "email": "test.user21@enduser1.com"
  },
  "tag": "news"
}

`preset-context` Command

since 1.2.0

The preset-context command configures preset context, which supplies per-run variable overrides on top of the global context and controls the test flow. It exposes two mutually exclusive families:

variables manages the seed values used before any task runs.
flow lists, imports, or clears the test execution flow (the order and dependencies between tasks).

A single command call may use only one family. Mixed invocations are rejected.

preset-context <family> [args]

`variables` Operations

Operation	Arguments	Description
`list`	—	Display all current values, or a message if none are set
`set`	`key=value,...`	Set one or more key/value pairs (comma-delimited)
`delete`	`key,...`	Delete one or more keys by name (supports dotted paths)
`clear`	—	Clear all preset values; the flow is preserved
`help` / `h`	—	Show usage

`flow` Usage

Arguments	Description
`list`	Display the current flow entries, or a message if none are set
`flow.json`	Import a JSON file whose top-level object contains a `flow` array; other properties are ignored and the existing flow is replaced
`clear`	Clear the existing flow; preset values are preserved

See the Define Test Flow section for full details on authoring the flow file.

Examples

# List all preset values
docker exec -it waterwheel-agent preset-context variables list

# Set preset overrides
docker exec -it waterwheel-agent preset-context variables \
  set username=admin,password=123456789

# Set nested values with dotted keys
docker exec -it waterwheel-agent preset-context variables \
  set user.username=admin,user.password=123456789

# Remove keys (supports dotted keys)
docker exec -it waterwheel-agent preset-context variables \
  delete username,user.password

# Clear preset values (flow is preserved)
docker exec -it waterwheel-agent preset-context variables clear

# Import the test flow from a JSON file inside the container
docker exec -it waterwheel-agent preset-context flow /agent/import/preset-flow.json

# Clear the flow (preset values are preserved)
docker exec -it waterwheel-agent preset-context flow clear

# List current flow entries
docker exec -it waterwheel-agent preset-context flow list

Importing a flow file from the host

preset-context flow <file> resolves its path inside the container. Without a mounted volume, push the file in first with upload-instruction-file, then import it:

cat ./preset-flow.json | \
  docker exec -i waterwheel-agent upload-instruction-file preset-flow.json
docker exec -it waterwheel-agent preset-context flow /agent/instructions/preset-flow.json

Operation Rules

Key names are case-sensitive. We recommend using lowercase or camelCase names.
Values may be quoted or unquoted.
Dotted keys create nested objects (for example, user.username=admin becomes { "user": { "username": "admin" } }).
variables and flow are mutually exclusive in a single call.
flow list displays the entries currently stored in preset-context.json; if none are set, a message is shown.
When importing a flow file, the JSON must contain a top-level flow array; extra properties are ignored.
The flow file path is resolved inside the container, not on the host machine. Make sure the file is reachable from a mapped volume (for example, under /agent) before referencing it.
Unknown key names in delete print a warning but do not cause an error; any found keys are still deleted.

`reset-test-config` Command

since 1.3.0

reset-test-config deletes test task files and/or instruction files to return the agent to a clean state.

reset-test-config [-t] [-i]

Options

Option	Description
`-t`	Delete all markdown test task files
`-i`	Delete all instruction files except `email-permissions.yaml`

Behavior

When neither -t nor -i is provided, both operations are performed.
email-permissions.yaml is always preserved when resetting instructions.
Resetting instructions also turns off host testing, keeping the agent's reported state consistent.
Missing directories are reported but do not cause an error.

Examples

# Reset both tasks and instructions
docker exec -it waterwheel-agent reset-test-config

# Reset task files only
docker exec -it waterwheel-agent reset-test-config -t

# Reset instruction files only
docker exec -it waterwheel-agent reset-test-config -i

`run-qa` Command

since 1.0.0

run-qa executes all tests found under the /agent/tasks folder. It performs the following steps in order:

Clear outputs from any previous run
Restart the preinstalled MCP servers and reload any changed permissions
Start the AI agent to run tests and write results

run-qa

From the host
docker exec -it waterwheel-agent run-qa

Dry Run Mode

run-qa supports a --dry-run flag to help identify configuration issues without making any API calls.

docker exec -it waterwheel-agent run-qa --dry-run

In dry run mode, the command performs the following steps in order:

Clear outputs from any previous run
Restart the preinstalled MCP servers and reload any changed permissions
Start the AI agent to record the active configuration and prompts
Parse all test files under /agent/tasks and build a test plan

Where the dry run results go

Only the orchestrator's progress lines are printed to the terminal, ending with 🧪 Dry-run mode enabled. The configuration, prompts, and test summary are written to files, not to stdout:

Output	Location	Contents
Configuration and prompts	`/agent/outputs/agent.log`	The full system prompt and agent instructions the run would use, the resolved global variables, and a `[DryRun] N task(s) discovered.` line
Test plan	`/agent/outputs/test-plan.json`	One entry per discovered task — its `name`, source `file`, `id`, and a `status` of `queued`

Read them after the dry run completes:

# The parsed test plan
docker exec -it waterwheel-agent cat /agent/outputs/test-plan.json

# The task count the agent discovered
docker exec -it waterwheel-agent grep DryRun /agent/outputs/agent.log

test-plan.json
{
  "results": [
    {
      "name": "Test Wikipedia English Language Banner",
      "file": "test-wikipedia-english.md",
      "id": "3",
      "status": "queued"
    }
  ],
  "generated_at": "2026-07-29T07:26:20.005Z"
}

Each run clears the previous outputs

Step 1 wipes /agent/outputs, so test-plan.json from a dry run is removed the next time you call run-qa. Read it before starting the real run.

Inputs

The agent requires at least one test file under /agent/tasks. All test files must be text files with the .md extension.

Sample Test Task
Navigate to https://www.wikipedia.org
Click the English language link on the Wikipedia main page.
Confirm the banner text "Welcome to Wikipedia" is displayed on the English Wikipedia homepage.

See the Manage Test Tasks section for full details on authoring test files.

Only file extension matters

There is no required Markdown format for test file structure. Tasks are executed as long as the file extension is .md.

Outputs

After each run, the agent writes the following files to /agent/outputs.

File	Description
`agent.log`	AI agent execution log
`api-log.json`	LLM API call records and token usage. Only written when `ENABLE_API_LOGGING` is `true`
`test-results.json`	Summary of all test results for the run
`test-context.json`	All context values stored at the end of the run, such as AI-generated usernames and passwords
`<test-name>_log.json`	Per-test step log recording all AI-decided steps and tool calls
`<test-name>_audit.json`	Per-test audit log recording the tool calls the AI actually performed
`playwright.log`	Playwright MCP stdout log
`playwright.err`	Playwright MCP stderr log
`firewall.log`	Playwright firewall log. Only written when `FIREWALL_DEBUG` is `true`
`email-mcp.log`	Email MCP stdout log
`email-mcp.err`	Email MCP stderr log

`set-domain-permission` Command

since 1.3.0

set-domain-permission generates the browser domain allowlist from a comma-delimited list of domains. The agent can only navigate to whitelisted domains; if it tries to reach any other URL, the test fails immediately. This is the scripting-friendly alternative to configuring domain permissions through config-agent.

set-domain-permission [-l] <domain1,domain2,...>

Options

Option	Description
`-l`	Rewrite every `localhost` in the domains to `host.docker.internal`, useful when targeting a local dev server from inside Docker

Behavior

Domains are separated by commas. Quote any entry containing shell-special characters, for example "https://*.wikipedia.org".
Leading and trailing whitespace around each entry is trimmed.
Empty entries are ignored.
The allowlist is overwritten on each run.

Examples

# Set the allowlist from a list of domains
docker exec -it waterwheel-agent set-domain-permission \
  https://www.google.com,"https://*.wikipedia.org","http://localhost:8080"

# Rewrite localhost to host.docker.internal with -l
docker exec -it waterwheel-agent set-domain-permission \
  -l "http://localhost:8080","http://localhost:8025"

Quoting

The quotes are consumed by your host shell before the argument reaches the container. Keep any entry containing *, ?, or other shell-special characters quoted, exactly as shown above.

See the URL Permissions section for full details on configuring allowed domains.

`stop-qa` Command

since 1.1.0

stop-qa stops the currently tracked run-qa process tree, including the launched agent subprocess, if one exists. Because there can only be one run-qa process running in a container, stop-qa lets you stop the existing run-qa process in that container.

stop-qa

Example

docker exec -it waterwheel-agent stop-qa

`upload-instruction-file` Command

since 1.3.0

upload-instruction-file creates or replaces an instruction or config file (for example allowed-domains.yaml, email-permissions.yaml, or extra-instructions.md) using content read from stdin. This lets you push instruction files into the agent without editing files in place.

upload-instruction-file <filename>

Behavior

Content is read from stdin and written to the named instruction file.
Missing parent directories are created automatically.
An existing file is replaced, and a warning is printed when it is.

Examples

Content is piped in from the host, so use docker exec -i without -t:

# Create or replace allowed-domains.yaml from stdin
printf 'allowed:\n  - http://host.docker.internal:8080\n' | \
  docker exec -i waterwheel-agent upload-instruction-file allowed-domains.yaml

# Pipe a local file into the instructions folder
cat ./extra-instructions.md | \
  docker exec -i waterwheel-agent upload-instruction-file extra-instructions.md

`upload-test-task` Command

since 1.3.0

upload-test-task creates or replaces a markdown test task using content read from stdin. Only markdown files are accepted — the filename must end with .md.

upload-test-task <filename.md>

Behavior

Content is read from stdin and written to the named task file.
The filename must end with .md, otherwise the command fails.
Missing parent directories are created automatically.
An existing file is replaced, and a warning is printed when it is.

Examples

Content is piped in from the host, so use docker exec -i without -t:

# Create or replace a test task from stdin
printf '# Login test\n' | docker exec -i waterwheel-agent upload-test-task login.md

# Pipe a local file into the tasks folder
cat ./checkout.md | docker exec -i waterwheel-agent upload-test-task checkout.md

# Confirm the agent picked it up
docker exec -it waterwheel-agent manage-test-files list

Commands​

Running the Commands​

check-test-result Command​

Output​

Example​

config-agent Command​

Main Menu​

First Run (no mode configured)​

Returning Runs​

Option 1 — AI Provider Mode​

Option 2 — Config Domain Permissions​

Option 3 — Test Web App on Host​

config-ai-provider Command​

Options​

Behavior​

Examples​

delete-builtin-skills Command​

Options​

Behavior​

Examples​

delete-test-skills Command​

Options​

Behavior​

Examples​

display-ai-config Command​

Output​

Example​

display-test-skills Command​

Options​

Behavior​

Examples​

enable-test-on-host Command​

Behavior​

Example​

get-failure-detail Command​

Options​

Output when a failed test is found​

Output when no failure​

Examples​

load-test-skills Command​

Behavior​

Examples​

manage-global-constants Command​

Operations​

Examples​

manage-test-files Command​

Operations​

Examples​

output-context-variables Command​

Options​

Errors​

Example​

preset-context Command​

variables Operations​

flow Usage​

Examples​

reset-test-config Command​

Options​

Behavior​

Examples​

run-qa Command​

Dry Run Mode​

Where the dry run results go​

Inputs​

Outputs​

set-domain-permission Command​

Options​

Behavior​

Examples​

stop-qa Command​

Example​

upload-instruction-file Command​

Behavior​

Examples​

upload-test-task Command​

Behavior​

Examples​

Commands

Running the Commands

`check-test-result` Command

Output

Example

`config-agent` Command

Main Menu

First Run (no mode configured)

Returning Runs

Option 1 — AI Provider Mode

Option 2 — Config Domain Permissions

Option 3 — Test Web App on Host

`config-ai-provider` Command

Options

Behavior

Examples

`delete-builtin-skills` Command

Options

Behavior

Examples

`delete-test-skills` Command

Options

Behavior

Examples

`display-ai-config` Command

Output

Example

`display-test-skills` Command

Options

Behavior

Examples

`enable-test-on-host` Command

Behavior

Example

`get-failure-detail` Command

Options

Output when a failed test is found

Output when no failure

Examples

`load-test-skills` Command

Behavior

Examples

`manage-global-constants` Command

Operations

Examples

`manage-test-files` Command

Operations

Examples

`output-context-variables` Command

Options

Errors

Example

`preset-context` Command

`variables` Operations

`flow` Usage

Examples

`reset-test-config` Command

Options

Behavior

Examples

`run-qa` Command

Dry Run Mode

Where the dry run results go

Inputs

Outputs

`set-domain-permission` Command

Options

Behavior

Examples

`stop-qa` Command

Example

`upload-instruction-file` Command

Behavior

Examples

`upload-test-task` Command

Behavior

Examples