Rebuild recovery runbook

When this container is replaced, everything needed to bring the terminal tunnel back lives under /home/hatch (the persistent volume) and can be restored with one command. This document explains what breaks, what survives, and how recovery works.

What a rebuild does

A rebuild gives you a fresh root filesystem but keeps /home/hatch.

One-command recovery

~/workspace/bin/recover-after-rebuild.sh

The script is idempotent — safe to run any time, on a fresh container or a healthy one. It does two things:

1. Provision (only on a fresh root filesystem). Detected by the absence of

the sentinel file /etc/hatch-provisioned (a fresh /etc means a fresh container). Provisioning: - Re-applies the apt mirror fix via ~/workspace/bin/fix-apt-mirror.sh (the stock ubuntu.sources ships a dead mirror.cogentco.com URI that hangs apt-get update forever; the file is owned by nobody, which container-root cannot edit in place, so the script deletes and recreates it). - Runs apt-get update if the package lists are missing. - Installs openssh-client if ssh is missing (the tunnel dials out with ssh -R; openssh-server is not needed for the current design). - Installs python3 if missing (required by ssh-via-proxy). - Sanity-checks that the persistent pieces exist (ttyd binary, tunnel-up.sh, gcp-tunnel-up.sh, ssh-via-proxy, ~/.ttyd-pass, ~/.ssh/vm_to_gcp, the persistent muse authorized_keys copy) and re-marks them executable. - Installs openssh-server if missing (the GCP relay's local endpoint) and recreates the muse login user + its authorized_keys from the persistent copy — /home/muse lives on the ephemeral root overlay, only /home/hatch survives a rebuild. - Touches the sentinel so provisioning isn't repeated. 2. Ensure the tunnels are up (always). If the supervisors aren't running, it starts them detached with setsid/nohup (each under its own flock guard with an anchored pgrep pattern so the two supervisors can never be confused for each other): - tunnel-up.sh — the browser terminal: ensures the stack is listening — internal ttyd on 127.0.0.1:7682 (no auth, localhost-only) and the cookie-session auth proxy ttyd-auth-proxy.py on 127.0.0.1:7681. - gcp-tunnel-up.sh — the GCP jump-host relay (the shared front door): ensures the container's sshd is listening on port 22 and dials ssh -N -R 127.0.0.1:2224:localhost:22 -R 127.0.0.1:7681:localhost:7681 to the GCP VM (34.139.37.135, user super, key ~/.ssh/vm_to_gcp). Before each dial it sweeps stale remote listeners (fuser -k 2224/tcp 7681/tcp), because dead sessions linger on the GCP side and would otherwise block the rebind. Skipped with a warning if ~/.ssh/vm_to_gcp is missing. Caddy on the GCP VM serves the stable public URL https://muse-dev.online from the 7681 forward. The script prints a status summary when done.

After recovery, the public URL is unchanged: https://muse-dev.online (the URL lives on the GCP VM's static IP, so rebuilds and redials don't affect it).

Manual recovery (if the script can't run)

# 1. Fix apt and install the SSH client

~/workspace/bin/fix-apt-mirror.sh

apt-get update -qq

apt-get install -y -qq openssh-client   # skip if `command -v ssh` works



# 2. Launch the supervisor (GCP relay carries both SSH and the terminal)

setsid nohup ~/workspace/bin/gcp-tunnel-up.sh >/dev/null 2>&1 < /dev/null &

disown



# 3. The public URL is always https://muse-dev.online (stable)



# 4. Confirm the GCP relay rebound (ports 2224 and 7681 on the jump host)

# from the GCP VM: ss -tln | grep -E '2224|7681'

# or check ~/workspace/tunnel/gcp-tunnel.log for "tunnel established"

How the pieces fit

• ~/workspace/bin/ssh-via-proxy — ProxyCommand helper (pure python):

carries the SSH session inside an HTTP CONNECT tunnel through the egress proxy. Reads proxy credentials from $HTTPS_PROXY. - (RETIRED 2026-10-02) ~/workspace/bin/tunnel-up.sh — the old localhost.run supervisor (ssh -R 80:localhost:7681 [email protected], URL in ~/workspace/tunnel/URL.txt). No longer started; kept in the repo only as a fallback. - ~/workspace/bin/ttyd-auth-proxy.py — cookie-session auth gate on 127.0.0.1:7681 (the only port the tunnel forwards to). Requires HTTP Basic once (user muse, password in ~/.ttyd-pass), then issues a signed HttpOnly; Secure session cookie; accepts cookie or Basic on every request including the /ws upgrade. Signing secret in ~/.ttyd-cookie-secret (auto-created, mode 600). This exists because iOS Safari does not resend cached Basic credentials on WebSocket upgrades — without it, iPhones load the page but the WS handshake is denied and the client loops on "Press Enter to Reconnect". - ~/workspace/bin/ttyd — the terminal server binary (1.7.7). Bound to 127.0.0.1:7682 with NO auth of its own (-W writable mode); it is only reachable via the proxy and must never be exposed directly. - ~/workspace/bin/ttyd-shell.sh — per-connection shell command: attaches to (or creates) tmux session main with TERM=xterm-256color exported (tmux refuses to start without a working terminfo entry). The tmux server outlives ttyd and the ssh tunnel, so reconnects resume the same shell. - ~/workspace/bin/gcp-tunnel-up.sh — supervisor for the GCP jump-host relay (the shared front door for native SSH, no browser needed). Keeps the container's sshd listening on port 22 and the ssh -N -R 127.0.0.1:2224:localhost:22 [email protected] session alive (via the egress proxy, key ~/.ssh/vm_to_gcp). The user connects with ssh -J super@<gcp-ip> -p 2224 muse@localhost. Dials go through the same 10–15 min proxy severing as the browser tunnel; before each (re)dial it runs fuser -k 2224/tcp on the GCP host to clear the stale listener left by the previous dead session (GCP sshd ClientAliveInterval 30 reaps ghosts in ~90 s as a backstop). The remote port must be unique per container dialing into the shared box — a second container needs its own port. Log: ~/workspace/tunnel/gcp-tunnel.log. - Container sshd + muse user — the relay's local endpoint. openssh-server is installed by recover-after-rebuild.sh; the muse account's authorized_keys live in /home/muse (persistent volume), so login keeps working across rebuilds. - ~/workspace/tunnel/tunnel.log / ssh-out.log — supervisor and ssh logs.

Automating detection (set up 2026-10-01)

The script handles how to recover; the runtime cron tunnel-watchdog (every 5 min) handles noticing. Each run executes ~/workspace/bin/recover-after-rebuild.sh — on a healthy container it's a no-op; after a rebuild it provisions and restores the tunnel unattended. Crons survive container rebuilds; VM-local processes don't.

The watchdog does double duty: it also owns public-URL health probing (see README "Health checking"), because the egress proxy password rotates per exec session and only the cron's fresh environment can probe reliably.

Note the public URL will have changed after any rebuild or forced redial — the new URL lands in ~/workspace/tunnel/URL.txt, and the user needs the new one (the watchdog reports it when a redial was forced; otherwise ask the assistant for the current URL).

Known gaps

• (RESOLVED 2026-10-02) The public URL used to change on every redial

(anonymous localhost.run tier). It is now stable — https://muse-dev.online via Caddy on the GCP VM — and the localhost.run tunnel is retired. - The egress proxy password rotates per exec session. Long-lived processes (the supervisor, ssh) keep working connections, but any NEW outbound connection with stale credentials silently hangs (no 407). Consequence: public-URL health probing must live in the fresh-environment watchdog cron, never in the supervisor — and if the ssh session itself ever needs to redial with very old credentials, the redial may hang; the watchdog forcing the redial is the recovery path. - openssh-server/sshd on port 22 (with the muse login user) IS part of recovery now — it's the local endpoint of the GCP relay. The old port-2222 sshd experiment is still excluded. - The GCP relay's remote ports (2224 for SSH, 7681 for the terminal proxy) must stay unique per container. When a second container dials into the same GCP box it needs its own ports and its own supervisor instance/key. - The old LocalTunnel experiment (~/workspace/ltunnel/) is superseded and intentionally excluded from recovery. - Rebuild validation (2026-10-01 ~18:43 EDT): a real rebuild WAS observed — supervisor, ttyd, and ssh all died, uptime reset. recover-after-rebuild.sh restored everything unattended. Two issues found and fixed: (1) two concurrent recovery runs double-started the supervisor — fixed with an flock guard in ensure_tunnel(); (2) tmux didn't get installed during provisioning (likely apt lock contention with the system replay) — installed manually; the script already provisions tmux on fresh containers.

Quick reference