# API Test Runner Guide

This guide complements `skills/tools/testing/api-test-runner/SKILL.md` with practical repo layout options and copy/paste commands. Install
`api-test` via `brew install nils-cli` (also provides `api-rest` and `api-gql`).

## Directory layouts

### Option A (recommended): `setup/` canonical layout

Why:

- Matches the `setup/rest` and `setup/graphql` conventions used by focused protocol calls
- Avoids coupling to any particular test framework’s directory structure

Note:

- `--suite <name>` searches `tests/api/suites/<name>.suite.json` first, then falls back to `setup/api/suites/<name>.suite.json`.

Typical structure:

```text
setup/
  api/
    suites/
      smoke.suite.json
  rest/
    endpoints.env
    tokens.env
    requests/
      health.request.json
  graphql/
    endpoints.env
    jwts.env
    operations/
      countries.graphql
      countries.variables.json
```

Run:

```bash
api-test run \
  --suite smoke \
  --out out/api-test-runner/results.json \
  --junit out/api-test-runner/junit.xml
```

### Option B: put everything under `tests/`

This is supported today.

Suite resolution note:

- `--suite <name>` searches `tests/api/suites/<name>.suite.json` (fallback: `setup/api/suites/...`).
- If your suite manifest lives outside those suites dirs (e.g. fixtures), use `--suite-file <path>` (or set `API_TEST_SUITES_DIR`).

Recommended structure:

```text
tests/
  api/
    suites/
      smoke.suite.json
  rest/
    endpoints.env          # optional (use --url or REST_URL instead)
    tokens.env             # optional (use ACCESS_TOKEN in CI instead)
    requests/
      health.request.json
  graphql/
    endpoints.env          # optional (use --url or GQL_URL instead)
    jwts.env               # optional (use ACCESS_TOKEN in CI instead)
    operations/
      countries.graphql
      countries.variables.json
```

#### Bootstrap (copy bundled templates, then rename/move)

```bash
mkdir -p tests
cp -R "$AGENT_HOME/skills/tools/testing/api-test-runner/assets/scaffold/setup/api" tests/
cp -R "$AGENT_HOME/skills/tools/testing/api-test-runner/assets/scaffold/setup/rest" tests/
cp -R "$AGENT_HOME/skills/tools/testing/api-test-runner/assets/scaffold/setup/graphql" tests/
```

Then edit your suite file to reference `tests/...` paths and set `configDir` defaults.

Example (minimal pattern):

```json
{
  "version": 1,
  "name": "smoke",
  "defaults": {
    "noHistory": true,
    "rest": { "configDir": "tests/rest" },
    "graphql": { "configDir": "tests/graphql" }
  },
  "cases": [
    { "id": "rest.health", "type": "rest", "request": "tests/rest/requests/health.request.json" },
    { "id": "graphql.countries", "type": "graphql", "op": "tests/graphql/operations/countries.graphql" }
  ]
}
```

Run:

```bash
api-test run \
  --suite smoke \
  --out out/api-test-runner/results.json \
  --junit out/api-test-runner/junit.xml
```

#### CI note (URLs/tokens)

In CI, prefer explicit env vars (and runtime login) instead of committing real secrets into `tests/**/tokens.env` / `jwts.env`:

- REST:
  - URL: `--url ...` or `REST_URL=...`
  - Auth (option A): `ACCESS_TOKEN=...` (Bearer token)
- GraphQL:
  - URL: `--url ...` or `GQL_URL=...`
  - Auth (option A): `ACCESS_TOKEN=...` (Bearer token)

Option B (recommended when JWTs expire): runtime login via a single JSON secret + suite `auth` block:

- Provide a GitHub Secret (default name): `API_TEST_AUTH_JSON`
- Add `auth` to your suite manifest (typically `tests/api/suites/*.suite.json`; or `setup/api/suites/*.suite.json` if using the `setup/`
  layout)
- The runner logs in once per profile and injects `ACCESS_TOKEN` per case (no token files needed in CI)

## GitHub Actions

The repo ships a working example workflow using the bundled smoke suite (local REST + GraphQL fixture):

- `.github/workflows/api-test-runner.yml`

The bundled smoke suite expects a local fixture on `http://127.0.0.1:43127`:

```bash
python3 setup/fixtures/httpbin/server.py --port 43127
```

Run it in a separate terminal (or background it with `&`) so the suite can connect.

To make CI runs easier to scan (especially for non-engineers), generate a small human-friendly summary from the results JSON:

- It prints to the job log
- It appends to `$GITHUB_STEP_SUMMARY` (Actions UI)
- It can write a `*.summary.md` file to upload as an artifact

If you want to run a committed `tests/` layout instead:

- Remove the “Bootstrap smoke suite” step
- Change the run command to use `--suite <suite>` (or `--suite-file tests/api/suites/<suite>.suite.json`)

### Matrix sharding (parallel by tags)

For large suites, split cases into shards by adding shard tags and running the suite in a matrix.

Tagging pattern (example):

- Add a stable tag for the environment (e.g. `staging`)
- Add a mutually exclusive shard tag for every case (e.g. `shard:0`, `shard:1`)

Workflow sketch:

```yaml
strategy:
  fail-fast: false
  matrix:
    shard: ["0", "1"]

steps:
  - name: Run suite shard
    env:
      AGENT_HOME: ${{ github.workspace }}
      API_TEST_AUTH_JSON: ${{ secrets.API_TEST_AUTH_JSON }}
    run: |
      api-test run \
        --suite my-suite \
        --tag staging \
        --tag "shard:${{ matrix.shard }}" \
        --out "out/api-test-runner/results.shard-${{ matrix.shard }}.json" \
        --junit "out/api-test-runner/junit.shard-${{ matrix.shard }}.xml"

  - name: Summarize shard results
    if: always()
    run: |
      api-test summary \
        --in "out/api-test-runner/results.shard-${{ matrix.shard }}.json" \
        --out "out/api-test-runner/summary.shard-${{ matrix.shard }}.md" \
        --slow 5
```

Notes:

- `--tag` uses AND semantics when repeated: a case must include all selected tags to run.
- Use per-shard output filenames (and per-shard artifacts) to avoid collisions.
