---
name: aws-deployment-state
description: Lumen deploy target pivoted to AWS t4g.small new-account Free Plan; project-side docs/scripts rewritten 2026-05-25
metadata: 
  node_type: memory
  type: project
  originSessionId: 4059c30a-7172-4501-9264-82e562516963
---

**As of 2026-05-25 (post-cleanup-loop):** Lumen is **LIVE in production** at `https://lumen.ahmedhobeishy.tech` on AWS EC2 t4g.small in `eu-central-1`. main branch tip is `4b09651` on both local and `origin/main` (pushed at end of cleanup loop). Three-round Codex + Claude cleanup loop landed `ad03435`, `eb4a9b7`, `4b09651` — both reviewers converged empty in round 4. Loop fixed: EIP race in user_data, hard-coded admin email in compute.tf, Redis broker eviction policy (allkeys-lru → noeviction), noop-embedder docstring inaccuracy, on-box recovery doc that referenced workstation-only Terraform state, plus added `check_embedding_provider` to prod_guards mirroring the LLM guard. Master is still untouched per user rule.

- **Domain**: Cloudflare DNS (subdomain of user's `ahmedhobeishy.tech` zone, A record → EIP `3.74.54.147`, DNS-only mode / proxy OFF)
- **TLS**: Caddy with real Let's Encrypt cert (issuer E7, valid 2026-05-25 → 2026-08-23)
- **ENV=production**, **LLM_PROVIDER=openai** against Groq (`api.groq.com/openai/v1`), **LLM_MODEL=llama-3.3-70b-versatile** — the H6 prod-boot guard passed (it explicitly rejects LLM_PROVIDER=noop in production, so a successful boot proves Groq is wired)
- **IAM access key** `AKIA_REDACTED_BEFORE_PUBLISH` whose secret passed through tool-call params during `aws configure set` has been **deleted** (verified via console + CLI returns `InvalidClientTokenId`). A new key needs creation if more terraform applies are needed.
- **Groq key**: created in Groq console as `lumen-prod`. Old `lumen-eval` key is unchanged.

Earlier in session: The user signed up for a new AWS Free Plan account (6 months, $100 starter + up to $100 more credits, no card, auto-closes 2026-11-25). All Oracle deployment artifacts removed from the repo; the user's separate Oracle journey continues out-of-band per [[oracle-deployment-state]].

## Why we pivoted

- Frankfurt A1 capacity stayed `out of host capacity` for 24h+ of polite 60s-cadence retries.
- PAYG upgrade unblocked the A1 core limit (4 → 16) but a `TenantCapacityExceeded` on the region-subscription cap blocked the Stockholm fallback.
- AWS t4g.small free-trial promo runs through Dec 31 2026 — perfect 6-month window for portfolio activation.
- Same ARM64 architecture → identical Docker images, identical compose stack, only the VM-acquisition step differs.

## Project-side changes (committed in `claude/romantic-mayer-ab2e85` worktree)

**Deleted:**
- `scripts/oracle-bootstrap.sh`
- `docs/deployment/oracle-vps.md`

**Created:**
- `scripts/aws-bootstrap.sh` — same shape as the Oracle one (non-root admin, sshd hardening with M1 authorized_keys guard, ufw, fail2ban, Docker, Compose v2) **plus** a 4 GB swapfile block for the 2 GB RAM constraint, plus EC2 metadata-aware public-IP detection.
- `docs/deployment/aws-vps.md` — 10-step runbook mirroring the Oracle one (signup → t4g.small launch → Elastic IP → hardening → Docker → secrets → boot → TLS → DNS → smokes → day-2 ops). Adds a "2 GB RAM tuning" block (Postgres shared_buffers=192MB, Redis maxmemory=64mb, Celery concurrency=1) and a "split deploy" appendix that pushes Next.js frontend to Vercel free if the box gets tight.

**Updated:**
- `README.md` — "Deploy it" section rewritten for AWS, status footer notes the pivot, H4 row in status table renamed `AWS t4g.small single-VM deploy runbook`.
- `docs/release/operator-activation-runbook.md` — Steps 1–3 rewritten for AWS Free Plan signup + t4g.small launch + `aws-bootstrap.sh`. Step 5 (MCP) and Step 6 (screencast) marked ✅ DONE. Variables table updated (`AWS_REGION`, `SSH_KEY_PATH` style now `.pem`).
- `CHANGELOG.md` — new "Deploy target pivot" entry at top of [Unreleased]; A4's historical entry annotated with "(later replaced by...)"; "Operator runbook (next steps)" rewritten with strikethroughs on already-done items.
- `docs/release/1.1.0-agentic-pr-body.md` — TL;DR and H4 block annotate the pivot; operator-checklist row "Oracle ARM VM live" → "AWS t4g.small VM live".
- `docs/release/known-issues-post-1.1.0.md` — KI-1 / KI-3 / KI-6 / KI-8 / KI-10 references updated from `oracle-bootstrap.sh` / `oracle-vps.md` / 24 GB A1 to `aws-bootstrap.sh` / `aws-vps.md` / 2 GB t4g.small.
- `.env.example` — line 111 comment cross-ref now `aws-vps.md`.

## What stays the same

The unmodified `docker-compose.prod.yml` (FastAPI + Celery worker + beat + Postgres-pgvector + Redis + MinIO + Caddy 2) runs identically on t4g.small (with the swapfile + tuning block), Oracle A1, and Hetzner CAX11. Migration off AWS at end of trial = "rerun the same runbook against the new ARM64 Ubuntu 24.04 box" — no Docker rebuild, no code change.

## Operator's remaining steps (from `docs/release/operator-activation-runbook.md`)

1. **Sign up for AWS Free Plan** ✅ done — user reported confirmation email
2. **Launch t4g.small EC2** — pending (Steps 2.1–2.4 of the runbook)
3. **Bootstrap + deploy** — pending (rsync repo, run `aws-bootstrap.sh`, fill `.env.production` incl. Groq key, `docker compose up`)
4. **`make eval`** — pending (mints the real tutor-score for README badge)
5. **MCP publish** ✅ done (`io.github.ahmedEid1/lumen` v1.1.0 live)
6. **Screencast** ✅ done (`docs/screencast/walkthrough.mp4` silent captioned)
7. **`make publish-rewrite`** — pending (push branch + open PR)

## What landed in the deploy session (commit `1dc7502`)

- **Terraform stack at `infra/aws/`** — single `terraform apply -auto-approve` provisions:
  - t4g.small EC2 (`i-0f0e66ee5a9b70099`) in eu-central-1, default VPC
  - 30 GB gp3 encrypted root (free-tier covered)
  - SG with 22/80/443 ingress
  - Ed25519 keypair (generated by `tls_private_key`, stored locally as `infra/aws/keys/lumen-prod.pem`)
  - Elastic IP `3.74.54.147` (preserved across `-replace` of the instance)
  - `user_data` runs `aws-bootstrap.sh` non-interactively at first boot — installs 4 GB swap, Docker, ufw, fail2ban (relaxed jail config so deploy bursts don't bite), creates the `lumen` admin user with NOPASSWD sudo
- **AWS CLI configured** as profile `lumen` (~/.aws/credentials) — does NOT shadow the user's existing default/env-var setup with the old `admin` account in 280595863415
- **TLS** — Caddy 2 auto-fetched a Let's Encrypt cert (issuer ZeroSSL ECC DV, valid 2026-05-25 → 2026-08-23) for the `<EIP>.nip.io` host
- **App** — `docker-compose.prod.yml` running api + worker + beat + db (Postgres+pgvector) + redis + s3 (MinIO) + web + proxy (Caddy). 9 published courses + demo user seeded
- **ENV=staging** for now — the H6 prod-boot guard rejects LLM_PROVIDER=noop in production. Flip to ENV=production in `.env.production` (on the box) the moment a real LLM key is in place
- **Real prod-boot bug fixes** mirrored back to repo:
  - `docker-compose.prod.yml`: CORS_ORIGINS wrapped as JSON array (pydantic-settings parses list[str] as JSON, the previous plain URL failed at boot), missing env passthroughs (WEB_BASE_URL, BADGES_*, LLM_*, EMBEDDING_*) added, ENV made overridable
  - `apps/frontend/next.config.ts`: `eslint.ignoreDuringBuilds=true` (the CI lint job is the source of truth; failing the prod build on warnings is duplicate enforcement)
  - `scripts/aws-bootstrap.sh`: non-interactive mode + relaxed fail2ban jail + `mkdir -p /run/sshd` before `sshd -t` (cloud-init runs before systemd creates the dir, breaks the bootstrap)
- **Smoke verified** (from outside AWS):
  - `https://3.74.54.147.nip.io/api/v1/health/live` → 200
  - `https://3.74.54.147.nip.io/api/v1/health/ready` → 200 with `{db: ok, redis: ok}`
  - `https://3.74.54.147.nip.io/` → 200, Next.js home renders

## Operator follow-ups — ALL DONE

1. ✅ IAM access key `AKIA_REDACTED_BEFORE_PUBLISH` deleted (verified — `aws sts get-caller-identity` with that key returns `InvalidClientTokenId`)
2. ✅ Groq key in place (`gsk_i2hU...SPF3` as `lumen-prod-2`); `LLM_PROVIDER=openai`, `LLM_MODEL=llama-3.3-70b-versatile`, `ENV=production` all set on the box
3. ✅ `make eval` ran end-to-end — `mean_overall = 2.4 / 5` across 30 items (Groq Llama 3.3 70B + Cloudflare Workers AI bge-small-en-v1.5 for retrieval, 45 chunks across 8 indexed courses)
4. ✅ Web healthcheck fixed (commit `9ef3f04` — `HOSTNAME=0.0.0.0` + node-based fetch healthcheck replacing the wget-that-wasn't-installed)

## How to apply when re-entering

- Read `docs/release/operator-activation-runbook.md` for the operator's step-by-step
- Read `docs/deployment/aws-vps.md` for VM-side detail
- The MCP server is already live; the screencast is already in the repo; only Steps 2–4 + 7 are external work the operator hasn't done yet
- See [[oracle-deployment-state]] for what's still running on the user's Oracle side (retry loop continues out-of-band)
- See [[active-goal]] for the broader release state
