BuildKite Notes
===============

to trigger build using REST API:

    curl -u "jamesdbloom@gmail.com" https://api.buildkite.com/v2/organizations/mockserver/pipelines/mockserver/builds \
      -X POST \
      -F "commit=HEAD" \
      -F "branch=master" \
      -F "message=First build :rocket:"

build step (as docker command):

    # run as interactive container (for debugging)
    docker run -v /var/lib/buildkite-agent/builds/$BUILDKITE_AGENT_NAME:/build -i -t mockserver/mockserver:build /bin/bash

    # check files in build folder (for debugging)
    docker run -v /var/lib/buildkite-agent/builds/$BUILDKITE_AGENT_NAME/mockserver:/build -w /build/mockserver -a stdout -a stderr mockserver/mockserver:build ls -lrt

    # actually run the build
    docker run -v /var/lib/buildkite-agent/builds/$BUILDKITE_AGENT_NAME/mockserver:/build -w /build/mockserver -a stdout -a stderr mockserver/mockserver:build /build/mockserver/scripts/local_quick_build.sh

determine killed reason:

    dmesg | grep -E -i -B100 'killed process'

AWS infrastructure:

Build agents run in an AutoScaling Group managed by a Lambda-based autoscaler.
There are two stacks — check the **current** stack first.
See `.opencode/skills/aws-investigation/SKILL.md` for full details.

    # authenticate (SSO, browser-based MFA)
    aws sso login --profile mockserver-build

Current: Terraform-managed (eu-west-2)
---------------------------------------

Managed by `terraform/buildkite-agents/`. ASG name is generated by Terraform.

    # find ASGs tagged with buildkite-mockserver
    aws autoscaling describe-auto-scaling-groups \
      --region eu-west-2 --profile mockserver-build \
      --query 'AutoScalingGroups[?contains(Tags[?Key==`Stack`].Value | [0], `buildkite-mockserver`)].{Name:AutoScalingGroupName,Desired:DesiredCapacity,Instances:Instances[*].{ID:InstanceId,State:LifecycleState}}'

    # list running EC2 instances (substitute ASG name from above)
    aws ec2 describe-instances \
      --filters "Name=tag:aws:autoscaling:groupName,Values=<ASG_NAME>" \
      --region eu-west-2 --profile mockserver-build \
      --query 'Reservations[].Instances[].{ID:InstanceId,State:State.Name,Launch:LaunchTime}'

    # check console output for a specific instance (for debugging boot issues)
    aws ec2 get-console-output --instance-id <instance-id> --region eu-west-2 --profile mockserver-build \
      --query 'Output' --output text

    # manually scale up agents if Lambda scaler is broken
    aws autoscaling set-desired-capacity \
      --auto-scaling-group-name "<ASG_NAME>" \
      --desired-capacity 2 --region eu-west-2 --profile mockserver-build

Legacy: CloudFormation-managed (us-east-1) — being replaced
-------------------------------------------------------------

    # check ASG status (instance count, health)
    aws autoscaling describe-auto-scaling-groups \
      --auto-scaling-group-names "buildkite-AgentAutoScaleGroup-VGG28FR0DE6Q" \
      --region us-east-1 --profile mockserver-build \
      --query 'AutoScalingGroups[0].{Desired:DesiredCapacity,Instances:Instances[*].{ID:InstanceId,State:LifecycleState}}'

    # manually scale up agents if Lambda scaler is broken
    aws autoscaling set-desired-capacity \
      --auto-scaling-group-name "buildkite-AgentAutoScaleGroup-VGG28FR0DE6Q" \
      --desired-capacity 2 --region us-east-1 --profile mockserver-build