# Release Process

> Read [release-principles.md](release-principles.md) first if you're modifying anything in this pipeline. The principles are load-bearing: ignore them and you'll re-create the tight CI coupling we just removed.

## Operator runbook

The end-to-end checklist a release manager follows. **Use this every release.** Everything below assumes a healthy `master` branch.

### 1. Decide the version

Run the `/prepare-release` slash command from this repo. It inspects `changelog.md`, `mockserver/pom.xml`, and the latest `mockserver-X.Y.Z` git tag, then recommends:

- `release-version` (e.g. `6.1.0`)
- `next-version` (e.g. `6.1.1-SNAPSHOT`)
- `old-version` (e.g. `6.0.0` — auto-derived, you don't need to type it on the form)
- `release-type` (almost always `full`)
- `create-versioned-site` (`yes` for major/minor, `no` for patch)

The skill applies SemVer rules:

| Trigger in `## [Unreleased]` | Bump |
|---|---|
| Any bullet prefixed `BREAKING:` | **Major** |
| Bullets under `### Added` or `### Changed` | **Minor** |
| Only `### Fixed` bullets | **Patch** |
| Empty/vague | **Block** — don't release |

If you want to override the recommendation, fine — but be deliberate about it.

### 2. Validate locally (optional but recommended)

```bash
./scripts/release/test-all.sh --quick
```

Runs every component in dry-run mode locally. Takes ~5 min (Maven Central / maven-plugin / javadoc are skipped under `--quick`). The full run (~25 min) is `./scripts/release/test-all.sh`. Working tree must be clean afterwards — if you see modifications, that's a bug in a dry-run, fix it before triggering CI.

### 3. Trigger a dry-run on Buildkite (recommended for major releases)

Open https://buildkite.com/mockserver/mockserver-release → "New Build" and fill in:

| Field | Value |
|---|---|
| Branch | `master` |
| Commit | (latest on master) |
| Release Version | from step 1 |
| Next SNAPSHOT Version | from step 1 |
| Release Type | `Full Release (all steps)` |
| Create Versioned Site? | `Yes` for major/minor, `No` for patch |
| **Dry Run?** | **`Yes — build/validate only, skip publish`** |

The dry-run exercises every step inside the actual Buildkite container images. Treat it as the final gate before publishing. It still requires the TOTP and downstream-approval block steps.

### 4. Trigger the real release

Same form as step 3, but flip **Dry Run? → `No — actually publish`**.

After the form, Buildkite immediately hits a `block` step asking for a 6-digit TOTP. The token expires every 30 s; if your agent fleet is cold-starting, you may have to wait ~1 min for an agent to come up before the TOTP step actually runs. The verifier accepts ±5 minutes of clock skew, so a slow start is forgiven.

### 5. Manual gate 1 — enter the TOTP

The TOTP seed lives in Secrets Manager under `mockserver-release/totp-seed`. Use the same authenticator app you set up for previous releases. If you've lost the seed, rotate it: generate a new seed (e.g. `python3 -c "import secrets, base64; print(base64.b32encode(secrets.token_bytes(20)).decode())"`), update the secret in AWS Secrets Manager, and re-enroll it in your authenticator (issuer `MockServer Release`).

After this gate the pipeline runs `Prepare` (pom bump + tag + push) and then `Maven Central` (mvn deploy + Sonatype publish + sync wait). Maven Central typically takes 15–25 min.

### 6. Watch Maven Central

Open these URLs in tabs while step 5's job runs:

- **Live deployment state at Sonatype:** https://central.sonatype.com/publishing/deployments
  - States transition: `VALIDATING` → `VALIDATED` → `PUBLISHING` → `PUBLISHED`
  - `FAILED` means the pipeline will abort and surface the reason
- **Canonical "is it live?" check** (returns 200 once synced): https://repo1.maven.org/maven2/org/mock-server/mockserver-netty/<release-version>/
  - The pipeline polls this URL itself; you can watch the same thing in your browser
- **Central artifact view** (what end users see): https://central.sonatype.com/artifact/org.mock-server/mockserver-netty/<release-version>

### 7. Manual gate 2 — approve downstream publish

After Maven Central is live, the pipeline pauses at a second `block` step labelled "Approve downstream publish". Sanity-check Maven Central one more time (the link above), then click **Unblock**.

Everything downstream now runs in parallel: Versioned Site, Maven Plugin, Docker, npm, Helm, Javadoc, SwaggerHub, Website, JSON Schema, PyPI, RubyGems, GitHub Release.

### 8. Verify the publishes

| Channel | Verification |
|---|---|
| Docker Hub | https://hub.docker.com/r/mockserver/mockserver/tags — `<release-version>`, `<release-version>-graaljs`, and `latest` should appear |
| npm — mockserver-node | https://www.npmjs.com/package/mockserver-node |
| npm — mockserver-client-node | https://www.npmjs.com/package/mockserver-client-node |
| PyPI | https://pypi.org/project/mockserver-client/ |
| RubyGems | https://rubygems.org/gems/mockserver-client |
| GitHub Release | https://github.com/mock-server/mockserver/releases |
| Helm chart | https://www.mock-server.com/index.yaml — should list the new version |
| Versioned docs site (major/minor only) | `https://<release-version-with-dash>.mock-server.com` — e.g. `6-1.mock-server.com` |
| Website | https://www.mock-server.com — version pin in the footer should match |
| Homebrew (a few hours later — bumped by BrewTestBot) | https://formulae.brew.sh/api/formula/mockserver.json → `.versions.stable` should equal `<release-version>` |

### 9. Homebrew — fully automated, no action required

The `mockserver` formula in `Homebrew/homebrew-core` is bumped automatically by **BrewTestBot** (Homebrew's own automation account). The chain:

1. The Maven release publishes a `mockserver-netty-<version>-brew-tar.tar` artifact to Maven Central (built and signed by `maven-central.sh`'s `-P release` profile, same lifecycle as the regular jars).
2. The Homebrew formula has a `livecheck` block pointing at Maven Search (`https://search.maven.org/remotecontent?filepath=org/mock-server/mockserver-netty/maven-metadata.xml`). BrewTestBot's scheduled livecheck picks up the new version, computes the URL + SHA256, and opens a PR against `Homebrew/homebrew-core`.
3. Homebrew CI builds bottles (pre-compiled binaries) for the supported macOS/Linux targets. A Homebrew maintainer reviews and merges the BrewTestBot PR once all checks pass.
4. End-users running `brew upgrade mockserver` get the new version.

The whole cycle from Maven Central publish → live on Homebrew typically takes a few hours. No human action is required, and no MockServer-side script needs to invoke `brew`. The only thing the release pipeline has to keep doing is publishing the `*-brew-tar.tar` artifact (see `mockserver/pom.xml`'s `release` profile).

If a bump ever does not happen within a day or two of release, check:
- The `mockserver-netty-<version>-brew-tar.tar` artifact is actually present at `https://repo1.maven.org/maven2/org/mock-server/mockserver-netty/<version>/` (canonical mirror) and resolvable via `https://search.maven.org/remotecontent?filepath=org/mock-server/mockserver-netty/<version>/mockserver-netty-<version>-brew-tar.tar` (the URL livecheck actually polls).
- The `mockserver.rb` formula in `homebrew-core` still has a `livecheck` block (`gh api repos/Homebrew/homebrew-core/contents/Formula/m/mockserver.rb`).
- BrewTestBot's recent activity on this formula: `gh search prs --repo Homebrew/homebrew-core --author BrewTestBot mockserver`.
- Whether the BrewTestBot PR is open but unmerged: a Homebrew maintainer needs to approve and merge.

If a manual bump is genuinely required (e.g. the bot is broken), `brew bump-formula-pr --strict --version=<release-version> mockserver` from a workstation with `brew` and an authenticated `gh` CLI will open the PR by hand.

### 10. Announce (optional)

If this is a notable release, post to:

- mockserver Slack / Discord (if you have one)
- The `mock-server` GitHub Discussions / Releases page (the GitHub Release notes are auto-generated from the changelog by the `github.sh` component)
- Twitter / Mastodon / etc.

---

## Architecture

```
scripts/release/
├── _lib.sh                       # shared functions: logging, dry-run, AWS,
│                                 # git, docker wrapper, version helpers
├── release.sh                    # orchestrator (prepare → update-version-references → components → finalize)
├── prepare.sh                    # validate + bump pom + tag + push
├── update-version-references.sh  # commit + push changelog / _config.yml /
│                                 # package.json / etc. so the parallel
│                                 # publish group reads the new version
├── finalize.sh                   # SNAPSHOT bump + deploy to Sonatype
├── preflight.sh                  # verify host has docker + bash + git + jq …
└── components/                   # one script per deployable artifact
    ├── maven-central.sh          # build + sign + Sonatype + publish + wait
    ├── maven-plugin.sh           # mockserver-maven-plugin release
    ├── docker.sh                 # multi-arch Docker Hub + ECR Public
    ├── npm.sh                    # mockserver-node + mockserver-client-node
    ├── pypi.sh                   # mockserver-client-python
    ├── rubygems.sh               # mockserver-client (Ruby)
    ├── helm.sh                   # Helm chart (OCI: GHCR + legacy HTTP: S3)
    ├── javadoc.sh                # Javadoc to S3
    ├── website.sh                # Jekyll site
    ├── schema.sh                 # JSON Schema
    ├── swaggerhub.sh             # OpenAPI spec to SwaggerHub
    ├── github.sh                 # GitHub Release
    └── versioned-site.sh         # X-Y.mock-server.com Terraform

.buildkite/scripts/
├── release-runner.sh             # Buildkite adapter (meta-data → env vars)
└── release-verify-totp.sh        # Buildkite-only TOTP gate

.buildkite/release-pipeline.yml   # flat list of steps; each step is one
                                  # release-runner.sh invocation
```

## How a release happens

### Step-by-step, locally

```bash
# 1. Verify your machine has the required host tools.
./scripts/release/preflight.sh

# 2. Run the entire pipeline in dry-run mode. Builds everything, but skips
#    every external write (npm publish, twine upload, S3 sync, gh release
#    create, git push, etc.).
./scripts/release/release.sh --version 6.1.0 --dry-run

# 3. Run a single component.
./scripts/release/components/npm.sh --dry-run        # exits with `RELEASE_VERSION` unset
RELEASE_VERSION=6.1.0 ./scripts/release/components/npm.sh --dry-run

# 4. Run only a few components.
./scripts/release/release.sh --version 6.1.0 --only=npm,pypi --dry-run

# 5. Skip components.
./scripts/release/release.sh --version 6.1.0 --skip=docker --dry-run
```

DRY_RUN defaults to `true` unless you pass `--execute`. **Locally you almost never want `--execute`** — that publishes for real.

### Step-by-step, on Buildkite

The same scripts run; the only difference is the wrapper:

1. Operator triggers the `mockserver-release` pipeline.
2. The input step collects: release version, next SNAPSHOT, type, versioned-site flag.
3. The TOTP block prompts for a 6-digit code; `release-verify-totp.sh` validates it.
4. Each subsequent step calls `.buildkite/scripts/release-runner.sh <stage>`, which:
   1. Reads Buildkite meta-data and exports it as `RELEASE_VERSION`, `NEXT_VERSION`, etc.
   2. Sets `DRY_RUN=false` (Buildkite releases for real).
   3. `exec`s the matching script under `scripts/release/`.

The release scripts themselves see only env vars. They have no idea Buildkite exists.

## Disaster recovery

Buildkite outage on release day? No problem. From a developer machine with `docker`, `aws`, `git`, `jq`, `python3`, and `bash`:

```bash
# Authenticate to AWS (for Secrets Manager + S3)
aws sso login --profile mockserver-build

# Run the same scripts the CI would have run
RELEASE_VERSION=6.1.0 \
NEXT_VERSION=6.1.1-SNAPSHOT \
RELEASE_TYPE=full \
CREATE_VERSIONED_SITE=yes \
./scripts/release/release.sh --execute
```

Every component runs in the same pinned Docker image whether you're on a laptop or a CI agent. There is no implicit CI state to recreate.

## Switching CI providers

This pipeline assumes nothing about Buildkite. If you want to run it on GitHub Actions, write a 30-line `.github/workflows/release.yml` that:

1. Receives release inputs (`workflow_dispatch` with `release_version` etc.).
2. Exports them as env vars.
3. Calls `./scripts/release/release.sh --execute` (or one component at a time, with explicit job dependencies).

Same for any other CI provider. The release scripts don't change.

## The contract

Release scripts (`scripts/release/*`) read these env vars:

| Variable | Required | Default | Purpose |
|---|---|---|---|
| `RELEASE_VERSION` | yes | — | The version being released (X.Y.Z) |
| `NEXT_VERSION` | no | `RELEASE_VERSION` patch+1 -SNAPSHOT | Next dev version |
| `OLD_VERSION` | no | latest `mockserver-X.Y.Z` tag | Previous release |
| `RELEASE_TYPE` | no | `full` | One of: full, maven-only, docker-only, post-maven |
| `CREATE_VERSIONED_SITE` | no | `no` | `yes` for major/minor releases |
| `DRY_RUN` | no | `true` | `false` to actually publish |
| `AWS_PROFILE` | no | (not set) | Used outside CI for Secrets Manager auth |

No other vars are read. **No `BUILDKITE_*` lookups happen in release scripts** — that's the whole point.

## Version-bearing files updated by the release pipeline

The release pipeline writes every version-bearing file in the repo, so contributors should not maintain version numbers by hand. The list is exhaustive — if a release ever leaves one of these files stale, it is a pipeline bug.

| File | Updated by | What gets set |
|------|------------|---------------|
| `mockserver/pom.xml` and every child pom (~34 files — the full mockserver/ subtree, excluding `target/`) | `prepare.sh` (`update_pom_versions` in `_lib.sh`) | `<version>` and `<parent><version>` from `SNAPSHOT` → `RELEASE_VERSION` |
| `mockserver/pom.xml` and child poms (re-bump) | `finalize.sh` (`update_pom_versions`) | `<version>` and `<parent><version>` from `RELEASE_VERSION` → `NEXT_VERSION` (the next `-SNAPSHOT`) |
| `changelog.md` | `update-version-references.sh` | Promote `## [Unreleased]` to `## [RELEASE_VERSION] - YYYY-MM-DD` and re-open an empty `## [Unreleased]` |
| `jekyll-www.mock-server.com/_config.yml` | `update-version-references.sh` | `mockserver_version`, `mockserver_api_version`, `mockserver_snapshot_version` |
| `mockserver/mockserver-core/src/main/resources/org/mockserver/openapi/mock-server-openapi-embedded-model.yaml` | `prepare.sh` | OpenAPI `version:` field |
| `mockserver-node/package.json` | `update-version-references.sh` | `version`, and the embedded `mockserver-netty-<version>-jar-with-dependencies.jar` URL |
| `mockserver-client-node/package.json` | `update-version-references.sh` | `version`, and `devDependencies["mockserver-node"]` |
| `mockserver-client-python/pyproject.toml` | `update-version-references.sh` | `version = "…"` |
| `mockserver-client-ruby/lib/mockserver/version.rb` | `update-version-references.sh` | `VERSION = '…'` |
| `mockserver-client-ruby/README.md` | `update-version-references.sh` | All occurrences of the old version literal |
| `helm/mockserver/Chart.yaml` | `components/helm.sh` | `version:` and `appVersion:` (must match app version per Helm policy) |
| All `*.html`, `*.md`, `*.yaml`, `*.yml`, `*.json`, `*.txt` outside `target/`, `node_modules/`, `helm/charts/`, `.tmp/`, and the changelog | `update-version-references.sh` (general find-and-replace) | Old version literal → new version literal; old API version → new API version |
| `terraform/website/terraform.tfvars` | `components/versioned-site.sh` | Append `"<MINOR>.<PATCH>" = { bucket_name = "…" }` and update `latest_version = "<SUBDOMAIN>"` |

**Sanity check before promoting a dry-run**: `git diff` should show every file in the table above changed exactly once. If `git diff --name-only | wc -l` is wildly larger than this table suggests, something is rewriting more than expected; if smaller, a version-bearing file may have been added without wiring it into `update-version-references.sh` (or `prepare.sh` for OpenAPI / pom files).

## Dry-run behaviour by component

| Component | Dry-run does | Dry-run skips |
|---|---|---|
| `prepare` | Validate inputs, show pom diff | pom write, git commit, tag, push |
| `maven-central` | `mvn clean install` (build + test) | Sonatype upload, publish, sync wait |
| `maven-plugin` | Build core + verify plugin | tag, deploy, push |
| `docker` | `docker buildx build` (local `--load`, amd64 only) | `--push` to Docker Hub + ECR |
| `npm` | `npm install`, grunt build | `git push tag`, `npm publish` (uses `--dry-run`) |
| `pypi` | `python -m build`, `twine check` | `twine upload` |
| `rubygems` | `gem build` | `gem push` |
| `helm` | `helm lint`, `helm package` | `helm push` to `oci://ghcr.io/mock-server/charts`, S3 upload, commit/push |
| `javadoc` | `mvn javadoc:aggregate` | S3 sync |
| `website` | `bundle install`, `jekyll build` | S3 sync, CloudFront invalidation |
| `schema` | jq-generate self-contained schemas | S3 sync |
| `swaggerhub` | Validate spec file | POST to SwaggerHub |
| `github` | Extract changelog notes, print preview | `gh release create` |
| `versioned-site` | `terraform plan` | `terraform apply`, S3 mirror |
| `update-version-references` | Show diff of version-reference rewrite | git push |
| `finalize` | Show pom version-bump diff | git push, mvn deploy snapshot |

## Pinned Docker images

All toolchain calls run inside these images. Defined in `scripts/release/_lib.sh`:

```bash
MAVEN_IMAGE=maven:3.9.9-eclipse-temurin-11
NODE_IMAGE=node:20-bookworm
RUBY_IMAGE=ruby:3.2-bookworm
HELM_IMAGE=alpine/helm:3.16.2
GH_IMAGE=maniator/gh:v2.62.0
PYTHON_IMAGE=python:3.12-slim-bookworm
TERRAFORM_IMAGE=hashicorp/terraform:1.9
```

Override any of them by exporting the corresponding env var. Change them in `_lib.sh` to update for everyone.

## Common operations

### Re-run a single component after a partial-pipeline failure

If, say, the Maven Central step succeeded but `npm` failed:

```bash
# On Buildkite: open the build, click Retry on the failed step. The
# release-runner.sh adapter re-reads meta-data and re-invokes.

# Locally:
RELEASE_VERSION=6.1.0 ./scripts/release/components/npm.sh --execute
```

### Reproduce a CI failure locally

```bash
# Pull the same env vars Buildkite was using (or set them by hand) and run
# the same script.
RELEASE_VERSION=6.1.0 \
NEXT_VERSION=6.1.1-SNAPSHOT \
./scripts/release/components/maven-central.sh --dry-run
```

That reproduces what the agent was doing, in the same Docker image, on your laptop.

### Add a new deployable component

1. Create `scripts/release/components/<name>.sh` following the pattern of an existing component.
2. Wire it into the orchestrator: add `<name>` to `ALL_COMPONENTS` in `release.sh`.
3. Add a step to `.buildkite/release-pipeline.yml` that runs `.buildkite/scripts/release-runner.sh <name>`.
4. Test with `RELEASE_VERSION=X.Y.Z ./scripts/release/components/<name>.sh --dry-run`.

## For agents / LLMs reading this in a future session

If you're modifying this pipeline, **respect the principles** in [release-principles.md](release-principles.md). In particular:

- Do NOT add `buildkite-agent meta-data get` or any `BUILDKITE_*` env-var reads to a script under `scripts/release/`. If a release script needs information that currently comes from Buildkite meta-data, plumb it through as a regular env var via the adapter.
- Do NOT call any tool natively if it has an upstream Docker image. The whole pipeline relies on language toolchains being containerised so the agents stay minimal and the scripts stay portable.
- Do NOT introduce dynamic pipeline generation (the previous design did this and we explicitly removed it). The Buildkite YAML is meant to be flat and obvious.
- DO add `--dry-run` support to every new component. The smoke-test pattern (`RELEASE_VERSION=99.99.0 ./scripts/release/components/<name>.sh --dry-run`) is how operators sanity-check changes locally before triggering CI.
- DO write each component as one self-contained file: build, package, sign, publish — all in one place. No splitting across multiple steps.

When in doubt, ask: "could a human ship this release from their laptop with just `docker`, `aws`, `git`, and `bash` installed?" If the answer is no, you've broken a principle.
