Rebuild recovery runbook
When this container is replaced, everything needed to bring the terminal tunnel back lives under /home/hatch (the persistent volume) and can be restored with one command. This document explains what breaks, what survives, and how recovery works — at the conceptual level.
Operators: the full version of this runbook, with this instance's
operational details (host IPs, ports, SSH users, key paths, exact
commands), is PIN-gated at/internal/recovery. Keep the two in sync
when the machinery changes.
Instance identity (domain, VM host, VM user) is read from config/white-label.yaml (FD_DOMAIN, FD_VM_HOST, FD_VM_USER via bin/wl-config.py --shell); the scripts fall back to this instance's values when the config is unavailable.
What a rebuild does #
A rebuild gives you a fresh root filesystem but keeps /home/hatch (including ~/hooks, where hook definitions and state live).
| Survives (persistent volume) | Lost (ephemeral root) |
|---|---|
~/workspace/ — all scripts, binaries, docs, logs, the .deb package cache, and persistent tool copies | /etc — apt sources, sshd config, SSH host keys, everything |
| Auth-proxy password and session-cookie signing secret (600) | apt-installed packages (SSH client/server, python3, tmux, age, …) |
~/.ssh/ — keys and client config | The relay's login user home (recreated on recovery) |
| Persistent SSH host keys (restored so clients don't see a new host key every rebuild) | All running processes (proxy, ttyd, ssh tunnel, supervisor) |
| Persistent copy of the relay login keys | /var/lib/apt/lists — package metadata |
~/hooks/ — hook definitions and state (the keeper's state survives) |
One-command recovery #
~/workspace/bin/recover-after-rebuild.sh
The script is idempotent — safe to run any time, on a fresh container or a healthy one. It does two things:
- Provision (only on a fresh root filesystem). Detected by the absence of
a sentinel file in /etc (a fresh /etc means a fresh container). Provisioning: - Re-applies the apt mirror fix (the stock Ubuntu sources ship a dead mirror URI that hangs apt-get update forever; the file is owned by nobody, which container-root cannot edit in place, so the script deletes and recreates it). - Installs packages from the persistent .deb cache when present — no network needed, which matters because the egress proxy is the slowest part of a rebuild — with per-package apt-get installs as fallback (SSH client for the outbound tunnels, SSH server for the relay's local endpoint, python3 for the proxy helper, tmux, age). - Restores the container's SSH host keys from the persistent store so the container keeps one stable host identity; seeds the store from a fresh install when empty. Must happen before sshd first starts. - Recreates the relay login user (its home is on the ephemeral root overlay), restores its authorized_keys from the persistent copy (the tunnel supervisor overwrites the live file from it every cycle — never edit the live file), and resets the documented login password. - Restores persistent tool copies and re-marks the persistent scripts executable. - Touches the sentinel so provisioning isn't repeated. 2. Ensure the tunnel is up (always). Under flock guards with anchored pgrep patterns so concurrent runs can never double-start a supervisor: - Makes sure the retired localhost.run supervisor and its ssh stay DOWN (the churny anonymous-tier tunnel is retired; a stable URL served from the jump-host VM replaced it). - Ensures the jump-host relay supervisor is running. It keeps the container's sshd up and dials the reverse-SSH session to the jump-host VM, sweeping stale remote listeners before each dial (dead sessions linger on the remote side and would otherwise block the rebind). - If the supervisor was (re)started, sends an immediate health report instead of waiting for the next scheduled one (see "Steady-state health"). - Prints a status summary.
After recovery, the terminal URL is unchanged — it lives on the jump-host VM's static IP, so rebuilds and redials don't affect it. (The URL itself is in the internal runbook.)
Automating detection (token-free hook) #
Recovery is owned by the tunnel-keeper hook — a runtime hook that polls every 60 s with a fresh environment per poll and no model involved, so it keeps working when token spend is exhausted and its egress-proxy credentials are always current (something a long-lived in-container supervisor can never guarantee). Hook definitions and state live in ~/hooks/ on the persistent volume, so the keeper itself survives rebuilds; the old tunnel-watchdog runtime cron is disabled (definition preserved) — it burned an agent run every 5 minutes for purely mechanical work.
Each poll:
- Runs the recover script (idempotent, bounded by timeout): re-provisions
after a rebuild, ensures the supervisor is up, keeps the retired tunnel down. 2. If the tunnel ssh has been absent for ≥2 consecutive polls, does the stale-listener sweep and dials with this poll's fresh env (the backstop dial — the supervisor's own redial is the first responder; this covers the case where its credentials went stale). 3. Probes the stable URL (fresh env ⇒ trustworthy): HTTP 200 or 401 = healthy (401 is the auth proxy asking for credentials — the tunnel serves). Two consecutive failures ⇒ kill the ssh pid(s) to force the supervisor to redial; the next poll redials if needed. 4. Chisel fallback (see docs/TRANSPORT-FALLBACK.md): if SSH has been down ≥3 polls but the HTTPS probe succeeded (the VM is reachable, the SSH path specifically is broken) and chisel credentials are provisioned, starts the chisel client carrying the same reverse forwards; stands it down when SSH recovers. 5. Silent when handled; wakes an agent only when something needs judgment (provisioning failed, or the tunnel stays down despite backstop dials). Wakes are rate-limited: at most one per 30 minutes per condition.
Steady-state health #
bin/health-report.sh (every 5 min via the health-reporter runtime cron) gathers local facts, signs them, and POSTs to the board's health ingest; the fleet health-report timer pattern is documented in docs/HEALTH.md. The recover script's immediate report on supervisor restart is only the fast-path for rebuilds — it never replaces the cron.
Manual recovery (if the script can't run) #
The exact commands (IPs, ports, users) are in the internal runbook. The shape is: fix the apt mirror, install the SSH/python/tmux/age packages, launch the relay supervisor detached, then confirm the jump-host relay rebound (its remote listener ports) from the VM side or the supervisor log.
How the pieces fit #
~/workspace/bin/ssh-via-proxy—ProxyCommandhelper (pure python):
carries the SSH session inside an HTTP CONNECT tunnel through the egress proxy. Reads proxy credentials from $HTTPS_PROXY. - (RETIRED) ~/workspace/bin/tunnel-up.sh — the old localhost.run supervisor. The recover script kills it if ever found running. Kept only as a fallback. - ~/workspace/bin/ttyd-auth-proxy.py — cookie-session auth gate in front of the terminal (the only port the tunnel forwards to). Requires HTTP Basic once, then issues a signed HttpOnly; Secure session cookie; accepts cookie or Basic on every request including the /ws upgrade. This exists because iOS Safari does not resend cached Basic credentials on WebSocket upgrades — without it, iPhones load the page but the WS handshake is denied and the client loops on "Press Enter to Reconnect". - ~/workspace/bin/ttyd — the terminal server binary. Bound to localhost with NO auth of its own (-W writable mode); it is only reachable via the proxy and must never be exposed directly. - ~/workspace/bin/ttyd-shell.sh — per-connection shell command: attaches to (or creates) a tmux session with TERM=xterm-256color exported. The tmux server outlives ttyd and the ssh tunnel, so reconnects resume the same shell. - ~/workspace/bin/gcp-tunnel-up.sh — supervisor for the jump-host relay (the shared front door for native SSH, no browser needed). Keeps the container's sshd listening and the reverse-SSH session alive via the egress proxy. The remote ports must be unique per container dialing into the shared box — a second container needs its own ports and its own supervisor instance/key. - Container sshd + login user — the relay's local endpoint, recreated by the recover script on every fresh container. - ~/workspace/bin/recover-after-rebuild.sh — the one-command recovery (this document's subject), run by hand and by the tunnel-keeper hook.
Known gaps #
- The egress proxy password rotates per exec session. Long-lived processes
(the supervisor, ssh) keep working connections, but any NEW outbound connection with stale credentials silently hangs (no 407). Consequence: health probing and the backstop dial must live in the fresh-environment hook, never in the supervisor. Old proxy creds often keep working through a grace period — don't rely on it. - The relay's remote ports must stay unique per container dialing into the same jump-host box. - The chisel fallback has its own fragilities — fingerprint rotation on server restart, the supervisor's stale-listener sweep killing the chisel server's listeners — documented in docs/TRANSPORT-FALLBACK.md. SSH remains the primary transport. - Rebuild validation: a real rebuild was observed — supervisor, ttyd, and ssh all died, uptime reset. The recover script restored everything unattended. Two issues found and fixed: concurrent recovery runs double-started the supervisor (fixed with a flock guard), and tmux wasn't installed during provisioning (apt lock contention; the script already provisions tmux on fresh containers).
Quick reference #
| Item | Location |
|---|---|
| Recovery script | ~/workspace/bin/recover-after-rebuild.sh |
| Tunnel keeper (live) | tunnel-keeper runtime hook, 60 s poll (~/hooks/scripts/tunnel-keeper.sh, state: ~/hooks/state/tunnel-keeper/) |
| Old watchdog (disabled) | tunnel-watchdog runtime cron — definition preserved, superseded by the hook |
| Relay supervisor | ~/workspace/bin/gcp-tunnel-up.sh (log: ~/workspace/tunnel/gcp-tunnel.log) |
| Full operational details | /internal/recovery (PIN-gated) |
| Health reports | health-reporter runtime cron (5 min) → bin/health-report.sh → board health ingest |
| Chisel fallback | docs/TRANSPORT-FALLBACK.md |
| ProxyCommand helper | ~/workspace/bin/ssh-via-proxy |
| ttyd binary | ~/workspace/bin/ttyd (localhost-bound, no auth of its own) |
| ttyd-auth-proxy (auth gate) | ~/workspace/bin/ttyd-auth-proxy.py |
| ttyd per-connection shell | ~/workspace/bin/ttyd-shell.sh (tmux session main) |
| Apt mirror fix | ~/workspace/bin/fix-apt-mirror.sh |
| Persistent .deb cache | ~/workspace/debs/ |
| Provision sentinel | /etc/hatch-provisioned (ephemeral — absence means "fresh") |