Usage

Commands

The container provides the following commands to help you run web tests.

Command	Usage
`check-test-result`	Check test results
`config-agent`	Configure AI provider, browser permissions, and access to a test website deployed on the host machine
`config-ai-provider`	Apply an AI provider and model without interactive prompts
`display-ai-config`	Show the effective AI provider, model, and token mode
`enable-test-on-host`	Enable testing against a web app on the host machine
`get-failure-detail`	Get failure details
`manage-global-constants`	Manage global constants
`manage-test-files`	Manage markdown test task files
`output-context-variables`	Output user-scoped context values from the latest run
`preset-context`	Manage preset context variables and test flow
`reset-test-config`	Delete test tasks and/or instruction files to reset state
`run-qa`	Run QA tasks
`set-domain-permission`	Set the browser domain allowlist from a domain list
`stop-qa`	Stop current QA run
`upload-instruction-file`	Upload an instruction or config file into the agent
`upload-test-task`	Upload a markdown test task into the agent

To run these commands on your host machine, prepend docker exec -it waterwheel-agent.

Execute run-qa on host machine
docker exec -it waterwheel-agent run-qa

`check-test-result` Command

since 1.1.0

check-test-result prints the overall outcome of the latest run, or reports the current run status if a test is still in progress. It is a helper command for code agent integration.

check-test-result

Output

Condition	Output
`run-qa` is currently active	A message indicating testing is in progress, including the orchestrator PID
Test results exist	The exit condition summarizing the run outcome
`test-results.json` missing, `agent.log` missing	`ℹ️ No test results found.`
`test-results.json` missing, `agent.log` exists	`ℹ️ No test results found.` followed by the full content of `agent.log`

`config-agent` Command

since 1.1.0

The config-agent command allows users to configure the AI provider, browser permissions, and access to a test website deployed on the host machine.

config-agent

First Run (no mode configured)

If no AI provider mode has been set yet, the script skips the main menu and goes directly into AI provider mode selection. The user must choose and confirm a mode before the main menu appears.

Returning Runs

========================================
  Waterwheel Agent Configuration
========================================

  Select an item to configure:

  1. AI provider mode [Anthropic Default Mode]
  2. Config domain permissions
  3. Test web app on host [disabled]
  0. Exit

----------------------------------------
  Choice:

The current state of option 1 (active mode name) and option 3 (enabled/disabled) is shown inline. Enter the number and press Enter.

Option 1 — AI Provider Mode

Selects the AI provider and model the agent uses. The AI provider must match the provider of your AI API key. When no provider mode is configured, the script enters initial mode automatically. Once a provider mode is set, the menu only shows modes belonging to the same provider as the current mode. The Manual customized mode is not shown.

Option 2 — Config Domain Permissions

This option allows users to configure the agent's domain access permissions. If the agent tries to access any URLs not whitelisted through domain permissions, the test fails immediately.

See the URL Permissions section for full details on configuring allowed domains.

Option 3 — Test Web App on Host

If the website under test is hosted on your local machine, for example at http://localhost:8080, enable this option so the agent in the Docker container can interact with it.

`config-ai-provider` Command

since 1.3.0

config-ai-provider applies an AI provider and model without the interactive prompts of config-agent. It is the scripting-friendly way to set up the AI provider, for example in CI or a one-off setup command.

config-ai-provider --provider <provider> --model <model> --mode <default|efficiency>

Options

Option	Description
`--provider <value>`	AI provider to use (for example, `anthropic` or `openai`). Must match the provider of your AI API key
`--model <value>`	Model name to use
`--mode <value>`	`default` for the standard prompt mode, or `efficiency` for the token-efficiency mode

Behavior

The provider must match the provider of your AI API key.
Provider locking applies: once a provider is configured, switching to a different provider requires a new container.
For provider-specific extras (such as a custom base URL or a localized system prompt), use the interactive config-agent command instead.

Examples

# Apply Anthropic default mode with an explicit model
config-ai-provider --provider anthropic --model claude-sonnet-4-5 --mode default

# Apply OpenAI token-efficiency mode
config-ai-provider --provider openai --model gpt-5.4 --mode efficiency

`display-ai-config` Command

since 1.3.0

display-ai-config prints the effective AI runtime settings as JSON. Use it to confirm which provider, model, and token mode the agent will use before running tests.

display-ai-config

Output

Returns a JSON object with exactly these keys:

Key	Description
`aiProvider`	The active AI provider
`aiModel`	The active model
`tokenMode`	`default` for the standard prompt mode, or `efficiency` for the token-efficiency mode. Empty when not configured

Example

display-ai-config

`enable-test-on-host` Command

since 1.3.0

enable-test-on-host enables testing against a web app deployed on your host machine, mirroring the Test web app on host action in config-agent. Use it when the website under test is hosted on your local machine, for example at http://localhost:8080, so the agent inside the Docker container can reach it.

When enabled, the command:

Instructs the agent to rewrite localhost to host.docker.internal during test runs.
Rewrites localhost to host.docker.internal in your domain allowlist.
Rewrites every localhost value in your global constants to host.docker.internal.

enable-test-on-host

Behavior

Re-running is idempotent: host testing is not enabled twice if it is already enabled.
The domain allowlist and global constants are only rewritten if they exist and contain localhost.

Example

enable-test-on-host

`get-failure-detail` Command

since 1.1.0

When a test failure is reported by a previous run, this command prints a full diagnostic report for the first failed test found in $AGENT_PATH/outputs/test-results.json. If a test run is still in progress, it reports that instead. It is another helper command for code agent integration.

get-failure-detail [-d]

Options

Option	Description
`-d`	Include API log (`/agent/outputs/api-log.json`) at the end of the report if file exists

Output when a failed test is found

Each section is printed in order. Missing files are reported inline and do not abort the output.

Section	Source
Failed Test Summary	The failed test JSON object from `test-results.json`
Test Detail	`/agent/tasks/<test-file>`
Test Steps	`/agent/outputs/<test-file-stem>_log.json`
Test Context	`/agent/outputs/test-context.json`
Agent Log	`/agent/outputs/agent.log`
API Log (only with `-d`, only if file exists)	`/agent/outputs/api-log.json`

Output when no failure

Condition	Output
`run-qa` is currently active	A message indicating testing is in progress, including the orchestrator PID
`test-results.json` missing, `agent.log` missing	`ℹ️ No test results found.`
`test-results.json` missing, `agent.log` exists	`ℹ️ No test results found.` followed by the full content of `agent.log`
Run did not complete	`⚠️ Run did not complete:` followed by the exit condition
Test results exist, no failed tests	`✅ No failed tests found in test results.`

`manage-global-constants` Command

since 1.1.0

The manage-global-constants command provides a simple entry point to configure global context environment variables used by tests.

manage-global-constants <operation> [args]

Operations

Operation	Arguments	Description
`list`	—	Display all current values, or a message if none are set
`set`	`KEY=value,...`	Set one or more key/value pairs (comma-delimited)
`delete`	`KEY,...`	Delete one or more keys by name (comma-delimited)
`clear`	—	Delete the entire context file
`help` / `h`	—	Show usage

Examples

# List all values
manage-global-constants list

# Set multiple values (quoted and unquoted)
manage-global-constants set BASE_URL="https://staging.example.com",TENANT=acme,SUPPORT_EMAIL=qa@example.com

# Set nested values with dotted keys
manage-global-constants set user.username=qa_user,user.password=secret

# Overwrite an existing key
manage-global-constants set TENANT=newcorp

# Delete specific keys (unknown keys produce a warning, known keys are still deleted)
manage-global-constants delete BASE_URL,TENANT

# Remove all values (also deletes the file)
manage-global-constants clear

Operation Rules

Key names are case-sensitive. We recommend using uppercase names.
Values may be quoted or unquoted.
Dotted keys create nested objects (for example, user.username=qa_user becomes { "user": { "username": "qa_user" } }).
Unknown key names in delete print a warning but do not cause an error; any found keys are still deleted.

`manage-test-files` Command

since 1.2.0

The manage-test-files command manages the markdown test task files the agent runs. Use it to add, list, and remove .md task files without copying files into the container by hand.

manage-test-files <operation> [args]

Operations

Operation	Arguments	Description
`list`	—	List `.md` task files with 1-based indexes
`add`	`path1,path2,...`	Add markdown files from file paths and direct directory children; non-markdown files are ignored
`delete`	`selector1,selector2,...`	Delete by 1-based index or exact filename (basename only), best effort
`clear`	—	Delete all markdown test files
`help` / `h`	—	Show usage

Examples

# List current markdown test files
manage-test-files list

# Add two markdown files
manage-test-files add ./instructions/test-a.md,./instructions/test-b.md

# Add all direct markdown files from a directory (subdirectories are ignored)
manage-test-files add ./instructions

# Delete by index
manage-test-files delete 1,3

# Delete by filename and index in one call
manage-test-files delete login-flow.md,2

# Clear all markdown test files
manage-test-files clear

Operation Rules

Only .md files are listed and managed.
add overwrites an existing file with the same destination name silently.
Directory import includes only direct child files, not subdirectories.
delete is best effort: invalid selectors print warnings while valid selectors are still deleted.
Source paths passed to add are resolved inside the container, not on the host machine. Make sure the files are reachable from a mapped volume (for example, under /agent) before referencing them.

`output-context-variables` Command

since 1.3.0

output-context-variables prints context values produced by the latest run as a flat JSON object. Tests store values such as AI-generated usernames and passwords in /agent/outputs/test-context.json; this command extracts those entries so you can reuse them outside the container (for example, to log in with a freshly registered account).

output-context-variables

Options

Option	Description
`-h`, `--help`, `h`, `help`	Show usage help

Errors

Condition	Result
`run-qa` is currently active	Prints an error that exporting context variables while testing is in progress isn't supported, and exits non-zero
`test-context.json` is missing	Prints an error that no `test-context.json` was found, and exits non-zero

Example

output-context-variables prints:

{
  "testuser": {
    "username": "test.user21",
    "password": "123456789Ab#",
    "email": "test.user21@enduser1.com"
  },
  "tag": "news"
}

`preset-context` Command

since 1.2.0

The preset-context command configures preset context, which supplies per-run variable overrides on top of the global context and controls the test flow. It exposes two mutually exclusive families:

variables manages the seed values used before any task runs.
flow lists, imports, or clears the test execution flow (the order and dependencies between tasks).

A single command call may use only one family. Mixed invocations are rejected.

preset-context <family> [args]

`variables` Operations

Operation	Arguments	Description
`list`	—	Display all current values, or a message if none are set
`set`	`key=value,...`	Set one or more key/value pairs (comma-delimited)
`delete`	`key,...`	Delete one or more keys by name (supports dotted paths)
`clear`	—	Clear all preset values; the flow is preserved
`help` / `h`	—	Show usage

`flow` Usage

Arguments	Description
`list`	Display the current flow entries, or a message if none are set
`flow.json`	Import a JSON file whose top-level object contains a `flow` array; other properties are ignored and the existing flow is replaced
`clear`	Clear the existing flow; preset values are preserved

See the Define Test Flow section for full details on authoring the flow file.

Examples

# List all preset values
preset-context variables list

# Set preset overrides
preset-context variables set username=admin,password=123456789

# Set nested values with dotted keys
preset-context variables set user.username=admin,user.password=123456789

# Remove keys (supports dotted keys)
preset-context variables delete username,user.password

# Clear preset values (flow is preserved)
preset-context variables clear

# Import the test flow from a JSON file
preset-context flow ./instructions/preset-flow.json

# Clear the flow (preset values are preserved)
preset-context flow clear

# List current flow entries
preset-context flow list

Operation Rules

Key names are case-sensitive. We recommend using lowercase or camelCase names.
Values may be quoted or unquoted.
Dotted keys create nested objects (for example, user.username=admin becomes { "user": { "username": "admin" } }).
variables and flow are mutually exclusive in a single call.
flow list displays the entries currently stored in preset-context.json; if none are set, a message is shown.
When importing a flow file, the JSON must contain a top-level flow array; extra properties are ignored.
The flow file path is resolved inside the container, not on the host machine. Make sure the file is reachable from a mapped volume (for example, under /agent) before referencing it.
Unknown key names in delete print a warning but do not cause an error; any found keys are still deleted.

`reset-test-config` Command

since 1.3.0

reset-test-config deletes test task files and/or instruction files to return the agent to a clean state.

reset-test-config [-t] [-i]

Options

Option	Description
`-t`	Delete all markdown test task files
`-i`	Delete all instruction files except `email-permissions.yaml`

Behavior

When neither -t nor -i is provided, both operations are performed.
email-permissions.yaml is always preserved when resetting instructions.
Resetting instructions also turns off host testing, keeping the agent's reported state consistent.
Missing directories are reported but do not cause an error.

Examples

# Reset both tasks and instructions
reset-test-config

# Reset task files only
reset-test-config -t

# Reset instruction files only
reset-test-config -i

`run-qa` Command

since 1.0.0

run-qa executes all tests found under the /agent/tasks folder. It performs the following steps in order:

Clear outputs from any previous run
Restart the preinstalled MCP servers and reload any changed permissions
Start the AI agent to run tests and write results

run-qa

Dry Run Mode

run-qa supports a --dry-run flag to help identify configuration issues without making any API calls.

run-qa --dry-run

In dry run mode, the command performs the following steps in order:

Clear outputs from any previous run
Restart the preinstalled MCP servers and reload any changed permissions
Start the AI agent to print the active configuration and prompts
Parse all test files under /agent/tasks and print a test summary

Inputs

The agent requires at least one test file under /agent/tasks. All test files must be text files with the .md extension.

Sample Test Task
Navigate to https://www.wikipedia.org
Click the English language link on the Wikipedia main page.
Confirm the banner text "Welcome to Wikipedia" is displayed on the English Wikipedia homepage.

See the Manage Test Tasks section for full details on authoring test files.

Only file extension matters

There is no required Markdown format for test file structure. Tasks are executed as long as the file extension is .md.

Outputs

After each run, the agent writes the following files to /agent/outputs.

File	Description
`agent.log`	AI agent execution log
`api-log.json`	LLM API call records and token usage. Only written when `ENABLE_API_LOGGING` is `true`
`test-results.json`	Summary of all test results for the run
`test-context.json`	All context values stored at the end of the run, such as AI-generated usernames and passwords
`<test-name>_log.json`	Per-test step log recording all AI-decided steps and tool calls
`<test-name>_audit.json`	Per-test audit log recording the tool calls the AI actually performed
`playwright.log`	Playwright MCP stdout log
`playwright.err`	Playwright MCP stderr log
`firewall.log`	Playwright firewall log. Only written when `FIREWALL_DEBUG` is `true`
`email-mcp.log`	Email MCP stdout log
`email-mcp.err`	Email MCP stderr log

`set-domain-permission` Command

since 1.3.0

set-domain-permission generates the browser domain allowlist from a comma-delimited list of domains. The agent can only navigate to whitelisted domains; if it tries to reach any other URL, the test fails immediately. This is the scripting-friendly alternative to configuring domain permissions through config-agent.

set-domain-permission [-l] <domain1,domain2,...>

Options

Option	Description
`-l`	Rewrite every `localhost` in the domains to `host.docker.internal`, useful when targeting a local dev server from inside Docker

Behavior

Domains are separated by commas. Quote any entry containing shell-special characters, for example "https://*.wikipedia.org".
Leading and trailing whitespace around each entry is trimmed.
Empty entries are ignored.
The allowlist is overwritten on each run.

Examples

# Set the allowlist from a list of domains
set-domain-permission https://www.google.com,"https://*.wikipedia.org","http://localhost:8080"

# Rewrite localhost to host.docker.internal with -l
set-domain-permission -l "http://localhost:8080","http://localhost:8025"

See the URL Permissions section for full details on configuring allowed domains.

`stop-qa` Command

since 1.1.0

stop-qa stops the currently tracked run-qa process tree, including the launched agent subprocess, if one exists. Because there can only be one run-qa process running in a container, stop-qa lets you stop the existing run-qa process in that container.

stop-qa

`upload-instruction-file` Command

since 1.3.0

upload-instruction-file creates or replaces an instruction or config file (for example allowed-domains.yaml, email-permissions.yaml, or extra-instructions.md) using content read from stdin. This lets you push instruction files into the agent without editing files in place.

upload-instruction-file <filename>

Behavior

Content is read from stdin and written to the named instruction file.
Missing parent directories are created automatically.
An existing file is replaced, and a warning is printed when it is.

Examples

# Create or replace allowed-domains.yaml from stdin
printf 'allowed:\n  - http://host.docker.internal:8080\n' | \
  upload-instruction-file allowed-domains.yaml

# Pipe a local file into the instructions folder
cat ./extra-instructions.md | upload-instruction-file extra-instructions.md

`upload-test-task` Command

since 1.3.0

upload-test-task creates or replaces a markdown test task using content read from stdin. Only markdown files are accepted — the filename must end with .md.

upload-test-task <filename.md>

Behavior

Content is read from stdin and written to the named task file.
The filename must end with .md, otherwise the command fails.
Missing parent directories are created automatically.
An existing file is replaced, and a warning is printed when it is.

Examples

# Create or replace a test task from stdin
printf '# Login test\n' | upload-test-task login.md

# Pipe a local file into the tasks folder
cat ./checkout.md | upload-test-task checkout.md

Commands​

check-test-result Command​

Output​

config-agent Command​

Main Menu​

First Run (no mode configured)​

Returning Runs​

Option 1 — AI Provider Mode​

Option 2 — Config Domain Permissions​

Option 3 — Test Web App on Host​

config-ai-provider Command​

Options​

Behavior​

Examples​

display-ai-config Command​

Output​

Example​

enable-test-on-host Command​

Behavior​

Example​

get-failure-detail Command​

Options​

Output when a failed test is found​

Output when no failure​

manage-global-constants Command​

Operations​

Examples​

manage-test-files Command​

Operations​

Examples​

output-context-variables Command​

Options​

Errors​

Example​

preset-context Command​

variables Operations​

flow Usage​

Examples​

reset-test-config Command​

Options​

Behavior​

Examples​

run-qa Command​

Dry Run Mode​

Inputs​

Outputs​

set-domain-permission Command​

Options​

Behavior​

Examples​

stop-qa Command​

upload-instruction-file Command​

Behavior​

Examples​

upload-test-task Command​

Behavior​

Examples​

Commands

`check-test-result` Command

Output

`config-agent` Command

Main Menu

First Run (no mode configured)

Returning Runs

Option 1 — AI Provider Mode

Option 2 — Config Domain Permissions

Option 3 — Test Web App on Host

`config-ai-provider` Command

Options

Behavior

Examples

`display-ai-config` Command

Output

Example

`enable-test-on-host` Command

Behavior

Example

`get-failure-detail` Command

Options

Output when a failed test is found

Output when no failure

`manage-global-constants` Command

Operations

Examples

`manage-test-files` Command

Operations

Examples

`output-context-variables` Command

Options

Errors

Example

`preset-context` Command

`variables` Operations

`flow` Usage

Examples

`reset-test-config` Command

Options

Behavior

Examples

`run-qa` Command

Dry Run Mode

Inputs

Outputs

`set-domain-permission` Command

Options

Behavior

Examples

`stop-qa` Command

`upload-instruction-file` Command

Behavior

Examples

`upload-test-task` Command

Behavior

Examples