# Framework Tests

Integration tests that verify varlock works correctly with real framework build pipelines. Each test creates an isolated temporary project, installs packed varlock packages from source, runs a real build, and asserts on the output.

## Running tests

From this directory (`framework-tests/`):

```sh
# Run all framework tests
bun run test

# Run tests for a specific framework
bun run test:astro
bun run test:expo
bun run test:nextjs

# Watch mode (re-runs on file changes)
bun run test:watch

# Re-pack varlock packages after making source changes
bun run repack
```

Or from the repo root:

```sh
bun run --filter varlock-framework-tests test
```

## How it works

### Test harness (`harness/`)

The shared harness provides `FrameworkTestEnv`, which manages the full lifecycle:

1. **Pack** — varlock packages are built and packed into `.tgz` tarballs (cached in `.packed/`; run `bun run repack` to refresh after source changes)
2. **Setup** — a temp project is created in `.test-projects/`, deps are installed via pnpm
3. **Scenario** — template files are copied, a build command runs, and output is asserted
4. **Teardown** — temp project is removed (set `KEEP_TEST_DIRS=1` to preserve for debugging)

### Adding a new framework

Create a directory under `frameworks/<name>/` with:

```
frameworks/<name>/
  <name>.test.ts          # Test file using FrameworkTestEnv
  files/
    _base/                # Files copied into every scenario (config, build scripts, etc.)
    schemas/              # .env.schema and env override files
    pages/                # Page/component templates swapped per scenario
```

If the framework needs a new varlock integration package, register it in `harness/pack.ts`.

### Test scenarios

Each scenario uses `describeScenario()` which:
- Copies template files into the project (merging fixture defaults with scenario overrides)
- Runs a build command (auto-prefixed with the package manager)
- Creates individual vitest tests for each `fileAssertion` and `outputAssertion`

### Environment variables

| Variable | Description |
|---|---|
| `KEEP_TEST_DIRS` | Set to `1` to preserve temp project dirs after tests |
| `REPACK` | Set to `1` to force re-packing varlock tarballs (otherwise cached) |

## Current frameworks

- **Astro** — tests static builds and SSR dev server, verifying env injection, leak detection in static output / client scripts / server pages / API endpoints, log redaction, and env vars in astro config
- **Next.js** — tests multiple versions (14, 15, 16) and bundlers (webpack, turbopack), verifying env injection, leak detection, log redaction, and sourcemap scrubbing
- **Expo** — tests the babel plugin transform pipeline, verifying static replacement of public vars, protection of sensitive vars, and correct handling of server (+api) routes

## Real deployment tests (`deploy/`)

Not part of normal test runs. These deploy the nextjs smoke app to a throwaway
Vercel project (**remote** build) and assert behavior that local tests
structurally can't reach: SSR on a real lambda (no varlock binary at runtime),
middleware on the real Edge runtime, runtime leak detection in production, and
log redaction inside Vercel's build + runtime log pipelines.

They run weekly (and via manual dispatch) through
`.github/workflows/deploy-tests.yaml` — failures are informational and never
block PRs. Auth dogfoods varlock: `deploy/.env.schema` resolves `VERCEL_TOKEN`
from 1Password (service account in CI via the existing `OP_CI_TOKEN` secret,
desktop app auth locally), so run them wrapped in `varlock run`:

```bash
cd framework-tests
VERCEL_DEPLOY_TESTS=1 varlock run --path ./deploy -- bunx vitest run deploy/
```

(Without the wrapper, the helpers fall back to your `vercel login` — but that
targets your account's scope rather than the shared varlockdev project.)

| Variable | Description |
|---|---|
| `VERCEL_DEPLOY_TESTS` | Required gate — tests are skipped without it |
| `DEPLOY_TESTS_PUBLISHED` | Test latest published packages instead of workspace code |
| `VERCEL_TOKEN` / `VERCEL_ORG_ID` / `VERCEL_PROJECT_ID` | Resolved by `deploy/.env.schema`; identify the varlockdev-owned project |

The Vercel project (`varlock-deploy-test`, under the varlockdev account) is a
throwaway with fake env values only; deployment protection is disabled on it so
routes can be asserted without SSO.

### Debugging real-deployment failures

Hard-won techniques (see `deploy/` helpers for implementations):

- **Edge failures**: don't iterate via deploys — Vercel runtime logs give
  messages without stacks. Run `vercel build` locally, then execute the exact
  deployed artifact in an edge-light-faithful VM with full stack traces:
  `node --experimental-vm-modules deploy/edge-sim.ts .vercel/output/functions/middleware.func/index.js --env _VARLOCK_ENV_KEY=... --path /middleware-test`.
  The sim encodes real edge-light behavior: no node:crypto (via
  `getBuiltinModule` OR `require`), read-only-ish `process.env`, eager
  microtask module instantiation, `_ENTRIES` handler registration shapes.
- **Silent remote build exits**: `next build` swallows type-check setup errors
  (`.catch(() => process.exit(1))` in next's type-check.js) — a missing
  `typescript`/`@types/node` dep dies with zero output right after "Skipping
  validation of types". Surface it by shipping a script that patches the catch
  to log (run via a temporary `vercel.json` `buildCommand`), or run
  `vercel build` locally where hoisting differences don't apply.
- **Runtime logs**: use the runtime-logs REST API with bounded reads + marker
  polling (`captureRuntimeLogs`) — streaming `vercel logs` is unreliable under
  vitest and the pipeline can lag by tens of seconds.
- **Failed deployments are kept** (passing ones are removed) — inspect them
  with `vercel inspect --logs <url>` and by re-requesting routes.
