Skip to main content

Advanced Usage

To fully customize the Waterwheel agent in a Docker environment, you can use Docker Compose to bring up the container using the instructions in this document.

These instructions work on macOS / Linux and Windows (PowerShell). Only the folder-setup script in step 1 differs between platforms; use the tabs to pick your shell. All docker compose commands are identical everywhere.

Installation​

note

All files mentioned in this section are bundled in waterwhell-installer.zip. Download it and extract its contents before proceeding.

1. Create the Folder Structure​

Run the helper script for your platform to create the local folder structure used for Docker volume mapping. If you omit the path, the current directory is used; if the path does not exist, it is created for you.

Use create-sync-subfolders.sh:

chmod +x create-sync-subfolders.sh
./create-sync-subfolders.sh /path/to/sync

This produces the directory layout that Docker Compose maps into the container:

<sync>/
└── agent/
    ├── instructions/   → /agent/instructions  (config files)
    ├── tasks/          → /agent/tasks          (your .md test files)
    └── outputs/        → /agent/outputs        (results and logs written by the agent)

Because these folders are bind-mounted, anything you place under agent/instructions on the host becomes the agent's runtime configuration. See Configuration Files for what each file does.

2. Configure Environment Variables​

Open dot_env.txt and fill in the following variables.

WATERWHEEL_SYNC_VOLUME_PATH=${to_fill} # The sync folder created in step 1 (e.g. /path/to/sync or C:\path\to\sync)
API_PROVIDER=${to_fill}                # AI provider
AI_MODEL=${to_fill}                    # AI model
AI_API_KEY=${to_fill}                  # AI API key

3. Place the Environment File​

Rename dot_env to .env and place it in the same directory as docker-compose.yml.

4. Copy Instruction Files​

Copy the following three files into agent/instructions under /path/to/sync.

You can further tune the run by adding the optional preset-context.json and extra-instruction.md files to the same folder. See Configuration Files for details.

5. Copy Task File​

Copy test-wikipedia-english.md into /path/to/sync/agent/tasks.

note

You can also try more complex chained test cases by copying 1_test_wikipedia_search.md and 2_test_wikipedia_navigate.md into /path/to/sync/agent/tasks. To enable the chaining, you must also copy preset-context.json into /path/to/sync/agent/instructions — its flow section defines the execution order and dependency between the two tasks. See Configuration Files for details. However, the API cost may be above $1 per test without configuration tuning.

6. Deploy the Docker Container​

From the same directory as docker-compose.yml, run:

docker compose up -d

Verify Installation​

To verify the installation, run the following command on your local machine and confirm no errors are reported in /path/to/sync/agent/outputs/agent.log.

docker compose exec waterwheel-agent run-qa --dry-run

Execute Tests​

To execute all tests under /path/to/sync/agent/tasks, run the following command on your local machine.

docker compose exec waterwheel-agent run-qa

Once the command completes, check the results at /path/to/sync/agent/outputs/test-results.json.

warning

OpenAI, even with its latest gpt-5.5 model, has difficulty differentiating between a link and its label. You may experience test failure if OpenAI is your configured AI provider. See our Provider Guide for details.

Configuration Files​

The files under agent/instructions (mapped to /agent/instructions in the container) control how the agent behaves. They are reloaded every time run-qa is called, so you can edit them on the host and simply start the next run — no container restart needed. The most commonly tuned files are described below.

global-context.json​

Static values shared across every test in the run — base URLs, tenant IDs, shared accounts, and similar constants. The values are injected into the agent's system prompt as global variables, so any test can reference them.

It is a plain JSON object and supports nested values via dotted keys.

global-context.json
{
    "TEST_URL": "https://www.wikipedia.org",
    "LOGIN_URL": "https://www.wikipedia.org/login",
    "user": {
        "username": "qa_user"
    }
}

You can edit this file directly, or manage it from inside the container with the manage-global-constants command. See Global Context for how tests consume these values.

preset-context.json​

Per-run configuration with two independent, optional sections:

  • data — seed values placed in the agent's context store before the first task runs. Unlike global context, these are not injected into the system prompt; the agent reads and overwrites them as tests run, which makes them useful for default credentials or feature flags a test can later change.
  • flow — the execution order of your tasks and the dependencies between them. Keeping the flow here (rather than in the .md files) lets you reuse the same task files across different suites and environments.
preset-context.json
{
    "data": {
        "admin_username": "qa_user",
        "admin_password": "s3cret",
        "feature_new_checkout": true
    },
    "flow": [
        { "file": "login.md", "node": 1 },
        { "file": "checkout.md", "node": 2, "required": [1] }
    ]
}

You can edit this file directly, or manage it with the preset-context command. See Preset context and Define Test Flow for full details.

allowed-domains.yaml​

The browser domain allowlist. The agent may only navigate to origins listed under allowed; if a test tries to reach any other URL, it fails immediately. This keeps test runs contained to the sites you intend to exercise.

allowed-domains.yaml
allowed:
  - https://*.wikipedia.org
  - http://host.docker.internal:8080

See URL Permissions for the matching rules and behaviour.

extra-instruction.md​

Free-form Markdown appended to the instructions for every test in the run. Use it for global rules that apply across all tasks.

Its most common use is testing a website hosted on your local machine. Inside the container, localhost refers to the container itself, not your host, so a locally hosted site is reached at host.docker.internal. Add a rewriting rule so the agent translates localhost URLs automatically:

extra-instruction.md
# Extra Test Instruction

## Global Rules

- **URL Rewriting:** Any URL containing `http://localhost` encountered during the
  test (in page content, emails, or links) MUST be rewritten to use
  `http://host.docker.internal` before navigating. For example,
  `http://localhost:8080/confirm-email?token=abc` becomes
  `http://host.docker.internal:8080/confirm-email?token=abc`. Never navigate to a
  `localhost` URL directly.

Make sure the corresponding host.docker.internal origin is also present in allowed-domains.yaml, otherwise the rewritten URL will be blocked.

tip

The config-agent command can generate and maintain extra-instruction.md for you when you enable Test web app on host.