1. Install prerequisites

pulumi up builds the frontend using pnpm, so you need Node.js and pnpm on your PATH.

You also need Docker running — the deploy builds container images.

Make sure you have at least ~20gb space free for the Pulumi stack.

brew install pulumi awscli uv python@3.13 jq node@22 pnpm

Or on Linux, install Pulumi, uv, the AWS CLI, Python 3.13+, jq, Node.js 22, pnpm, and Docker.

Required Settings

6. Deploy

Before your first deploy, make sure Docker Hub authentication is set up — the build pulls base images from Docker Hub, which rate-limits anonymous pulls:

docker login          # Docker Hub — required; anonymous pulls are rate-limited (https://hub.docker.com/)
docker login dhi.io   # Docker Hardened Images — Hawk's Python base lives here (free Community tier; same Docker Hub credentials work)

pulumi up

Secrets encryption (AWS KMS): With pulumi stack init ... --secrets-provider="awskms://alias/pulumi-secrets" (step 5), secret stack configuration is encrypted using KMS, not a passphrase. Do not set PULUMI_CONFIG_PASSPHRASE or rely on passphrase-based encryption for Hawk stacks.

KNOWN ISSUE — Pulumi prompts for a passphrase if --secrets-provider was omitted at pulumi stack init The project-level Pulumi.yaml ships with a hardcoded secretsprovider: pointing at a KMS alias most users don't have access to. If pulumi stack init was run without --secrets-provider, the stack inherits that default; Pulumi can't reach the key and falls back to prompting for a passphrase. Workaround: re-point the stack at your own KMS alias:

> pulumi stack change-secrets-provider "awskms://alias/<your-alias>"
> 

This rewrites the per-stack Pulumi.<stack>.yaml with the correct secretsprovider: line. Safe to run on a fresh stack with no resources yet. See Pulumi: changing secrets providers for context. This callout should be removed once Pulumi.yaml no longer ships with a real-looking default.

This creates roughly 200+ AWS resources including a VPC, EKS cluster, ALB, ECS services, Aurora PostgreSQL, S3 buckets, Lambda functions, and more. First deploy takes about 15-20 minutes.

KNOWN ISSUE — git-config secret placeholder is missing required keys The Pulumi-created secret <stack>/inspect/api-git-config has value {"GIT_CONFIG_COUNT": "0"}, but the API task definition references 7 JSON keys in it: GIT_CONFIG_COUNT, GIT_CONFIG_KEY_0..2, GIT_CONFIG_VALUE_0..2. ECS refuses to start a task whose secret reference points at a missing JSON key, so the API service enters deployment-circuit-breaker failure. Symptom: pulumi up succeeds but the API URL returns 503 with no healthy targets. The fix is a one-line change to the placeholder in infra/hawk/__init__.py. Until that lands, run one of the two workarounds below before continuing: Default — running only public evals (no GitHub auth needed): push an empty-but-structured secret. No PAT required:

> aws secretsmanager put-secret-value \
>   --secret-id <stack>/inspect/api-git-config \
>   --secret-string '{"GIT_CONFIG_COUNT":"0","GIT_CONFIG_KEY_0":"","GIT_CONFIG_VALUE_0":"","GIT_CONFIG_KEY_1":"","GIT_CONFIG_VALUE_1":"","GIT_CONFIG_KEY_2":"","GIT_CONFIG_VALUE_2":""}'
> 

If you need private GitHub repo cloning during evals (eval configs that reference packages via git+https://github.com/org/private-repo.git or git+ssh://git@github.com/...): run the helper script with a GitHub PAT:

> scripts/dev/set-git-config.sh <stack> <github-pat>
> 

Minimum permissions for a fine-grained PAT: - Resource owner: your user, or an org whose private repos the runner needs to clone - Repository access: the specific repos referenced in your eval configs (or "All repositories" within the owner) - Permissions → Repository permissions → Contents: Read-only Classic-PAT equivalent: repo scope (broader — also grants issues/PR access — but works). If every package you reference is in a public repo, don't bother with a PAT — the empty secret above is sufficient. Either way, force a new ECS deployment so the task picks up the corrected secret:

> aws ecs update-service --cluster <stack>-platform --service <stack>-hawk-api --force-new-deployment
> 

Wait ~60–90 seconds, then verify with aws ecs describe-services --cluster <stack>-platform --services <stack>-hawk-api --query 'services[0].[runningCount,deployments[0].rolloutState]' — should show runningCount: 1 and rolloutState: COMPLETED.

9. Install the Hawk CLI and run your first eval

```bash uv pip install "hawk[cli] @ git+https://github.com/METR/hawk#subdirectory=hawk"

Configure the CLI to point to your deployment

uv run python scripts/dev/generate-env.py <stack> > hawk/.env

uv run hawk login uv run hawk eval-set hawk/examples/simple.eval-set.yaml uv run hawk logs -f # watch it run uv run hawk web # open results in browser ```

What Hawk Deploys

When you run pulumi up, Hawk creates the following infrastructure on AWS:

The infrastructure is designed to scale down to near-zero cost when idle (Aurora auto-pauses, Karpenter scales EKS nodes to zero) and scale up automatically when you submit evaluations.

Managing Your Deployment

Quick Start

This gets you from zero to a working Hawk deployment on AWS. You'll need an AWS account and a domain name. You can use your existing OIDC identity provider for authentication, or a Cognito user pool by default.

KNOWN ISSUE — us-east-1 is currently broken; use a different region Two us-east-1-specific failure modes prevent Hawk from running there: (1) EKS doesn't support us-east-1e as a control-plane AZ, while Hawk's VPC uses all available AZs; (2) us-east-1 uses the legacy ec2.internal DNS suffix instead of <region>.compute.internal, which Bottlerocket's pluto doesn't accept — EKS nodes never join the cluster. Workaround: deploy to us-west-2 (project default, most-tested) or an EU region like eu-west-1 / eu-central-1. This warning should be removed once both underlying issues are fixed upstream.

Demo Video

5. Create and configure your stack

cd infra
pulumi stack init my-org --secrets-provider="awskms://alias/pulumi-secrets"
cp ../Pulumi.example.yaml ../Pulumi.my-org.yaml

Edit Pulumi.my-org.yaml with your values. At minimum, you need:

config:
  aws:region: us-west-2
  hawk:domain: hawk.example.com       # domain you control — used for API and service routing
  hawk:publicDomain: example.com       # parent domain for DNS zones and TLS certs
  hawk:primarySubnetCidr: "10.0.0.0/16"

That's enough to get started. The environment name defaults to your stack name.

Authentication: if you leave hawk:oidcClientId unset (the default), Hawk automatically provisions a Cognito user pool during pulumi up and wires it up as the auth provider. You'll create your first user in step 8 below using scripts/dev/create-cognito-user.sh.

Note that if you are using Cognito, then hawk login in step 9 requires browser authentication to complete the login flow. If you are deploying to a headless environment (such as a remote container) you will instead need to switch to the web viewer to continue, or setup your own authentication flow.

If you already have an OIDC provider (Okta, Auth0, etc.), use it instead (and skip step 8). Run the autodiscovery script to generate the config:

python scripts/dev/discover-oidc.py <issuer-url> <client-id> <audience>

Copy the output into your Pulumi.<stack>.yaml. See Pulumi.example.yaml for the full list of OIDC settings.

Configuration Reference

All configuration lives in Pulumi.<stack-name>.yaml. See Pulumi.example.yaml for a fully documented reference with all available options.

Infrastructure Options

Optional Integrations

These are all disabled by default. Enable them in your stack config when needed.

Datadog (monitoring, APM, log forwarding):

hawk:enableDatadog: "true"
hawk:datadogSite: datadoghq.com

Requires a <env>/platform/datadog-api-key secret in AWS Secrets Manager.

DNS / Route 53 / Cloudflare:

DNS is not optional — see Choose a domain and DNS strategy above for the four configuration paths (Route 53 Domains, manual delegation, Cloudflare automatic delegation, or HTTP-only testing mode).

Tailscale (VPN overlay for private service access):

Set hawk:albInternal: "true" and store a Tailscale auth key in AWS Secrets Manager. This makes all services accessible only through your Tailscale network.

Budget alerts:

hawk:budgetLimit: "10000"
hawk:budgetNotificationEmails:
  - "team@example.com"

When integrations are disabled, services fall back to simpler alternatives (CloudWatch instead of Datadog, no DNS delegation, etc.).

Eval Set Config

An eval set config is YAML that defines a grid of tasks, agents, and models. Hawk runs every combination.

tasks:
  - package: git+https://github.com/UKGovernmentBEIS/inspect_evals
    name: inspect_evals
    items:
      - name: mbpp

models:
  - package: openai
    name: openai
    items:
      - name: gpt-4o-mini

limit: 1 # optional: cap samples

Submit it:

hawk eval-set config.yaml

Multiple Environments

You can run multiple Hawk environments (staging, production, dev) from the same repo. Each gets its own Pulumi stack and isolated AWS resources.

```bash pulumi stack init staging

configure Pulumi.staging.yaml

pulumi up -s staging

pulumi stack init production

configure Pulumi.production.yaml

pulumi up -s production ```

Dev Environments

For development, you can create lightweight environments that share an existing stack's VPC, ALB, and EKS cluster while getting their own database and services:

./scripts/dev/new-dev-env.sh alice    # creates a dev-alice stack

Requires a deployed stg stack in the Pulumi backend (the script clones its config). Set PULUMI_BACKEND_URL and AWS_PROFILE first; see AGENTS.local.example.md for the env-var template.

Services appear at https://api-alice.hawk.<domain> and https://viewer-alice.hawk.<domain>. Tear down with:

pulumi destroy -s dev-alice
pulumi stack rm dev-alice    # only after destroy completes

4.2. Choose an API and VPC privacy level

4.2.1 Production environments - extra security

For production environments, we recommend for added security that your VPC is private (change hawk:albInternal: "true") and Hawk CLI on your development machine is used through Tailscale (see Tailscale under Optional Integrations below).

INFO: infra/hawk/api.py will automatically create A-alias records for api.hawk.<privateDomain> and middleman.<privateDomain> in your private hosted zone which are resolved through Tailscale DNS to access your API and middleman. Attempting to access either without Tailscale will result in DNS resolution errors.

4.2.2 Development environments - extra convenience

For development environments, your VPC can be public (the default) and Tailscale is optional. Authentication is still required for API and middleman functionality. Hawk CLI on your personal machine will access the API over the public internet.

INFO: if your VPC is public (the default hawk:albInternal: "false"), then infra/hawk/api.py will create additional A-alias records to your public hosted zone to resolve the DNS for api.hawk.<publicDomain> and middleman.hawk.<publicDomain>. This is for convenience during development.

- To add a public Route 53 alias record manually for api.hawk.<publicDomain> (and middleman.<publicDomain> if you'll hit it directly) pointing at the ALB. Get the ALB info from pulumi stack output alb_dns_name and pulumi stack output alb_zone_id, then in the public hosted zone for your publicDomain create A-alias records. Example:

>   AWS_PROFILE=<profile> aws route53 change-resource-record-sets \
>     --hosted-zone-id <public-zone-id> \
>     --change-batch '{ "Changes": [{ "Action": "UPSERT", "ResourceRecordSet": {
>       "Name": "api.hawk.<publicDomain>.", "Type": "A",
>       "AliasTarget": { "DNSName": "dualstack.<alb-dns>", "HostedZoneId": "<alb-zone-id>", "EvaluateTargetHealth": true } } }] }'
>   

7. Set up LLM API keys

Hawk routes model API calls through its built-in LLM proxy (Middleman). You need to provide at least one provider's API key:

scripts/dev/set-api-keys.sh <stack> OPENAI_API_KEY=sk-...

This stores the key in Secrets Manager and restarts Middleman. You can set multiple keys at once:

scripts/dev/set-api-keys.sh <stack> OPENAI_API_KEY=sk-... ANTHROPIC_API_KEY=sk-ant-...

<stack> is your Pulumi stack name (look it up with pulumi stack --show-name). Used consistently across steps 7–9 below. Supported providers: OpenAI, Anthropic, Gemini, DeepInfra, DeepSeek, Fireworks, Mistral, OpenRouter, Together, xAI.

KNOWN ISSUE — Middleman startup crashes if GCP project isn't set, even with no Vertex models in use Middleman's startup at middleman/src/middleman/server.py:136 unconditionally calls init_vertex_urls(), which requires GOOGLE_CLOUD_PROJECT_FOR_PUBLIC_MODELS or a project_id in GOOGLE_APPLICATION_CREDENTIALS_JSON. Symptom: Middleman tasks fail ALB health checks ("Target.Timeout" on port 3500); subsequent hawk eval-set calls return Middleman timeout. The proper fix is to make Vertex URL init lazy or gate it on whether any Vertex/Gemini model is configured. Workaround until fixed: set the Pulumi config to any value (a real GCP project if you use Gemini, or a sentinel like none otherwise) and re-run pulumi up:

> pulumi config set hawk:middlemanGcpProjectForPublicModels none
> pulumi up
> 

KNOWN ISSUE — Middleman's model registry is empty on standalone deploys hawk/hawk/tools/sync_models.py only does DB→DB sync (used by Pulumi for dev envs pointing at staging). For a fresh standalone deploy there's no source DB to sync from and no shipped default_models.json, so the registry stays empty. Symptom: hawk eval-set returns Middleman error: Models not found. The proper fix is for sync_models.py to accept a --from-json source plus a shipped default seed file in the repo. Workaround until fixed: insert at least one model manually via the RDS Data API:

> CLUSTER_ARN="arn:aws:rds:<region>:<account>:cluster:<stack>-inspect-ai-warehouse"
> SECRET_ARN='arn:aws:secretsmanager:<region>:<account>:secret:rds!cluster-<...>'  # single-quoted — Aurora's auto-created master secret has `!` in its name
>
> aws rds-data execute-statement \
>   --resource-arn "$CLUSTER_ARN" --secret-arn "$SECRET_ARN" --database inspect \
>   --sql "
>     WITH new_group AS (
>       INSERT INTO middleman.model_group (name) VALUES ('model-access-public')
>       ON CONFLICT (name) DO UPDATE SET name = EXCLUDED.name RETURNING pk
>     ),
>     new_model AS (
>       INSERT INTO middleman.model (name, model_group_pk)
>       SELECT 'claude-haiku-4-5', pk FROM new_group RETURNING pk
>     )
>     INSERT INTO middleman.model_config (model_pk, config, is_active)
>     SELECT pk,
>       jsonb_build_object('lab','anthropic','danger_name','claude-haiku-4-5',
>         'are_details_secret',false,'dead',false,'vision',false,
>         'max_tokens_keyword','max_tokens','request_timeout_minutes',30,'stream',false),
>       true FROM new_model RETURNING pk;
>   "
> 

Then force Middleman to reload: aws ecs update-service --cluster <stack>-platform --service <stack>-middleman --force-new-deployment. Naming gotcha: Middleman model groups use the prefix model-access-<name> (middleman/src/middleman/models.py:634). The user's JWT scope must match the group name exactly. For Cognito users, hawk:defaultPermissions defaults to "model-access-public" — so the model has to be in group model-access-public to be reachable (or you set hawk:defaultPermissions to grant another group). The SQL above already uses the right group name. Other valid lab values: openai, gemini, vertex, deepseek, mistral, xai, plus others — see middleman/src/middleman/models.py:32.