# kukeon

Build, test, and smoke-test conventions for anyone — human or agent — working on this repo.

## Conventions

- **License header:** Apache 2.0 + SPDX copyright header on every new Go file.
- **Shared error sentinels:** defined in `internal/errdefs/`. Don't declare new `errors.New` in package code when the concept is reusable — add it to `errdefs` instead.
- **Tests colocated with source:** `foo.go` + `foo_test.go`.

## The two default realms

`kuke init` provisions **two** realms, each mapped to its own containerd namespace (see `internal/consts/consts.go`):

| Realm         | Containerd namespace    | Purpose                                                      |
| ------------- | ----------------------- | ------------------------------------------------------------ |
| `default`     | `default.kukeon.io`     | User workloads. Created empty so `kuke create …` has a home. |
| `kuke-system` | `kuke-system.kukeon.io` | System workloads owned by kukeon itself.                     |

The `kukeond` daemon runs inside the **`kuke-system`** realm — specifically as a container inside the cell `kuke-system / kukeon / kukeon / kukeond` (realm / space / stack / cell). The `default` realm is deliberately left user-owned so `kuke purge --cascade` on it can never take down the daemon.

## CLI use-case reference

The full set of operator workflows the `kuke` CLI must support — each with its command sequence and behavioral invariants (exit codes, side effects, idempotency, error paths) — lives in [`docs/cli-use-cases.md`](docs/cli-use-cases.md). That document is workflow-oriented (not alphabetical by command) and is the source of truth for UX expectations the smoke test below does **not** cover, including image management, cell teardown verbs (`stop`/`kill`/`delete`/`purge`), and the cascade-purge / divergent-spec edge paths.

## Local smoke test: rebuild and re-run `kuke init`

When a change touches the build path, the daemon, or anything under `/opt/kukeon`, run this end-to-end before opening a PR.

### Prerequisites

`make dev-init` requires one daemon to be running on the host:

- **Standalone containerd** at `/run/containerd/containerd.sock` — used by `kuke build` and `kuke init`. This is _not_ the docker-private containerd at `/var/run/docker/containerd/containerd.toml`; `pgrep containerd` may show that one even when the system socket is missing. If `ls /run/containerd/containerd.sock` returns no such file, start it with `service containerd start` (or `systemctl start containerd`). On hosts with no init script and no systemd (e.g., the agent dev container), launch the binary directly: `containerd > /tmp/containerd.log 2>&1 &`.

No docker daemon is required. `kuke build` invokes the standalone `kukebuild` binary on the host — a BuildKit-as-library image builder that writes straight into the target realm's containerd namespace over the same `/run/containerd/containerd.sock`. `make dev-init`'s build phase produces `kukebuild` alongside `kuke` and places it on `PATH` (`/usr/local/bin`) so the `sudo ./kuke build` step resolves it; a missing `kukebuild` fails fast with `kuke build`'s "not found on PATH" message.

A containerd failure surfaces as a confusing error several phases into the script — `failed to connect to containerd: ... dial unix:///run/containerd/containerd.sock: timeout`. Bring it up before invoking `make dev-init` to skip the rabbit hole.

When run from inside a `kukeon-dev-root` cell (the canonical agent workflow), `make dev-init` automatically redirects the kukeond socket to `/run/kukeon-dev/` so it doesn't clobber the parent host's `kuke attach` plumbing — see [`docs/dev-init.md`](docs/dev-init.md) for the full bare-host vs. nested-mode contract.

### One-shot: `make dev-init`

```bash
make dev-init
```

`scripts/dev-init.sh` composes the full re-bootstrap loop: `make kuke kukebuild` + `kukeond` symlink, writes `./kukeond-dev.yaml` + `./kuke-dev.yaml` idempotently (guarded by `[ ! -e ]`) and threads `--server-configuration` (admin commands) + `KUKE_CONFIGURATION` (client + parity commands) through every invocation, `kuke daemon reset` of the prior cell, `kuke build -t kukeon.internal/kukeond:v0.0.0-dev --realm kuke-system .` building the image straight into the `kuke-system` realm's containerd namespace, `kuke init --kukeond-image kukeon.internal/kukeond:v0.0.0-dev`, and the daemon-parity check below. `kukeon.internal` is the ICANN-reserved, non-routable internal host the epic (#1063) adopts for every locally-built kukeon image — a stray pull of one fails fast against the reserved TLD instead of silently hitting Docker Hub. The published default (`KukeondImageRepo = ghcr.io/eminwux/kukeon` in `cmd/config/version.go`) is unchanged; only the local dev-loop ref moves. The script is idempotent — re-running on a healthy host produces a clean re-bootstrap.

The daemon-parity tail of the output (the regression guard) must read:

```
NAME         STATE  AGE  NAMESPACE
-----------  -----  ---  -------------------------
default      Ready  <age>  default.dev.kukeon.io
kuke-system  Ready  <age>  kuke-system.dev.kukeon.io
```

Both `kuke get realms -o wide` and `kuke get realms -o wide --no-daemon` must produce the same shape (same columns, same rows, same `STATE`, same `NAMESPACE`); `AGE` may differ by ms between the two invocations but renders at second granularity, so a single re-run normally produces byte-identical output. The tail above is the `KUKEON_PROFILE=dev` shape — export `KUKEON_PROFILE=dev` before `make dev-init` to get the dev profile (`*.dev.kukeon.io` namespaces, `-o wide` parity tail with the `NAMESPACE` column, `cgroupPath: /kukeon-dev/...`); this non-default profile is the canonical dev-init smoke since #285 phase 3. With `KUKEON_PROFILE` unset, the script deliberately keeps the historical default profile and the tail prints plain `kuke get realms` output (`NAME STATE AGE`, namespaces `*.kukeon.io`, `cgroupPath: /kukeon/...`) — same two-row parity contract, different columns and suffix. `CGROUP` no longer surfaces in either the default or wide table (epic:get retires it from both); the script's follow-up `kuke get realms -o yaml | grep cgroupPath` step prints `cgroupPath: /kukeon-dev/default` and `cgroupPath: /kukeon-dev/kuke-system` so the dev cgroup-root flip is visible too. If only the `--no-daemon` list is populated, the daemon's view of `/opt/kukeon` diverged from the in-process controller — that's the bind-mount regression this check catches.

**If your change touches anything the daemon reads, this check must be in the PR's test plan.** Reviewer agent will flag PRs that miss it.

After the daemon-parity tail, `scripts/dev-init.sh` also runs a PTY-driven `kuke attach` smoke against a kuketty-wrapped cell (`dev-init-attach/ds/dks/cattach`) — this exercises the attachable + kuketty socket pipeline end-to-end (post-create chown, `0o660 root:kukeon` socket mode, JSON-RPC + SCM_RIGHTS handshake, clean `^]^]` detach). Changes touching `internal/controller/runner/attachable.go` or any attach/kuketty socket path are validated by this phase; the no-op-StartCell wrong-mode reapply branch (#935/#936) is unreachable from CLI verbs (the higher-level `controller.StartCell` rejects already-running cells), so its specific correction behavior is covered exclusively by the unit tests in `internal/controller/runner/attachable_test.go`.

### Post-init: `kuke status`

`kuke status` is the canonical equivalent of the manual diff ritual above —
one command covering daemon liveness, host pre-flight, state consistency,
and the daemon-parity walk across every resource kind. See
[`docs/site/cli/kuke-status.md`](docs/site/cli/kuke-status.md) for the full
reference (sections, flag table, exit codes, JSON shape).

The two-line `kuke get realms` diff above stays the minimal pinned
regression guard the `make dev-init` tail prints; `kuke status` is the
broader smoke an operator runs after `kuke init` to surface anything the
two-realm tail won't catch (cgroup delegation gone, CNI binaries gone,
divergent secrets/blueprints/configs, residual containerd namespaces from
a half-cleaned `kuke uninstall`).

### Post-reboot cgroup recovery

A host reboot wipes the cgroup tmpfs while cell metadata under `/opt/kukeon/data/...` survives. The background reconciler (`(*Exec).ReconcileCell`) re-runs `ensureCellCgroup` on the next tick for any cell whose cgroup is absent AND whose persisted `Status.ReadyObserved` is true — re-creating the cgroup directory and re-asserting subtree controllers in the same pass.

Half-CreateCell cells (a `kuke create cell` that crashed before `markCellReady` could close the `ReadyObserved` latch) are deliberately excluded from the heal: the in-flight CreateCell will finish its own `ensureCellCgroup` path under the per-cell lock, and the reconciler stepping in would race that flow.

Consequence: after a reboot, no operator-driven `kuke start <name>` is required per cell — the next reconcile tick re-creates each previously-Ready cell's cgroup automatically. The `make dev-init` smoke does not exercise this path; reboot recovery is covered by the unit tests under `internal/controller/runner/reconcile_heal_test.go`.

### Manual phases (fallback)

To run individual phases by hand — e.g. while debugging a single phase — invoke them in order:

1. **Tear down the existing runtime.** `sudo ./kuke daemon reset` stops + deletes the prior `kukeond` cell and clears `/run/kukeon/kukeond.{sock,pid}`. User-realm data under `/opt/kukeon/data/default` is left intact, so `kuke purge --cascade` on `default` can never take down the daemon. Pass `--purge-system` to additionally wipe `/opt/kukeon/data/kuke-system` for a fully clean re-bootstrap.

2. **Build the binaries.**

   ```bash
   rm -f kuke kukeond kukebuild
   make kuke kukebuild # produces ./kuke and ./kukebuild
   ln -sf kuke kukeond # kukeond is argv[0]-dispatched from the same binary
   sudo ln -sf "$(pwd)/kukebuild" /usr/local/bin/kukebuild # kuke build resolves kukebuild via PATH
   ```

3. **Build the local `kukeond` image into the `kuke-system` realm (no registry push).**

   `kuke build` invokes `kukebuild` (BuildKit as a library), which writes the
   image straight into the realm's containerd namespace — no docker daemon and
   no `--from-docker` loader hop. The `kuke-system` realm must already exist
   (created by an earlier `kuke init` pass).

   ```bash
   sudo ./kuke build --build-arg VERSION=v0.0.0-dev -t kukeon.internal/kukeond:v0.0.0-dev --realm kuke-system .
   sudo ctr -n kuke-system.kukeon.io images ls | grep kukeon.internal/kukeond
   ```

4. **Run `kuke init`.**

   ```bash
   sudo ./kuke init --kukeond-image kukeon.internal/kukeond:v0.0.0-dev
   ```

   Expected tail:

   ```
       - cell "kukeond": created (image kukeon.internal/kukeond:v0.0.0-dev)
       - cell cgroup: created
       - cell root container: created
       - cell containers: started
   kukeond is ready (unix:///run/kukeon/kukeond.sock)
   ```

### Inspecting the running daemon

```bash
# Two bind mounts expected: /run/kukeon and /opt/kukeon (host→container, same path).
sudo ctr -n kuke-system.kukeon.io container info \
    kukeon_kukeon_kukeond_kukeond | \
    python3 -c "import sys,json; print(json.dumps([m for m in json.load(sys.stdin)['Spec']['mounts'] if m.get('type')=='bind'],indent=2))"

# Expected process line:
# /bin/kukeond serve --socket /run/kukeon/kukeond.sock --run-path /opt/kukeon
ps -ef | grep '[k]ukeond serve'
```

### Recovering from a failed `kuke uninstall`

If `kuke uninstall` reports `skipped (realm purge failed)` and `filesystem + user/group cleanup skipped: residual containerd namespace prevented teardown`, see the **Full per-host teardown** section in [`docs/cli-use-cases.md`](docs/cli-use-cases.md) for the half-cleaned-host gate explanation and the namespace-cleanup + re-run recovery procedure.
