INSTALL.md — R&D container networking setup (agent-facing)
Scope: replicate this container's SSH and terminal R&D stack on another, identical container. The container image is the same everywhere; only the per-install identity (keys, ports, secrets) changes.
The architecture, in one picture:
┌─ identical container ──────────────────────────────┐
│ ssh-via-proxy (CONNECT relay through egress proxy) │
│ gcp-tunnel-up.sh ── ssh -R 127.0.0.1:<PORT>:localhost:22
│ ttyd 1.7.7 + ttyd-auth-proxy.py + tmux ("main") │
│ tunnel-watchdog cron (every 5 min, runtime-side) │
└─────────────────────────┬──────────────────────────┘
│ dials OUT through the
│ egress proxy (rotating creds)
▼
shared GCP jump host (34.139.37.135)
user reaches the container via GCP
One rule that governs everything below: the GCP VM is the shared front door. New containers dial INTO it. There are no per-container localhost.run browser tunnels — that path exists only on the original container and is legacy. The GCP side will reverse-proxy browser traffic too; the container-side browser stack (ttyd, auth proxy, tmux) is the same regardless of which public path fronts it.
1. Before you start (prerequisites) #
You need from the human (or from the GCP owner):
- GCP VM address and SSH user — currently
34.139.37.135, usersuper. - Unique ports on the GCP VM for THIS container — check the registry
first: /home/super/machine-registry/PORTS.md on the GCP VM lists every taken port. You need one SSH port and one terminal port, both free (this container owns 2224/7681; the operator uplink super owns 2222). Two machines sharing a port will flap forever ("remote port forwarding failed"). 3. A fresh SSH keypair generated inside this container (ssh-keygen -t ed25519 -f ~/.ssh/vm_to_gcp -N ""). The PUBLIC key must be added to the GCP VM's super user authorized_keys. The private key never leaves the container. 4. Egress proxy env vars present (HTTPS_PROXY/https_proxy pointing at hatch-egress-proxy:3128). Verify: env | grep -i proxy. Without these, nothing below works. 5. A ttyd login password you generate yourself and hand to the user (user muse). Generate with openssl rand -hex 16.
Also confirm the container basics: /home/hatch is the only directory that survives a rebuild (/dev/mapper/rv, btrfs). /home/muse, /etc, apt packages, and all processes are wiped without warning. Everything durable must live under /home/hatch.
2. Files to bring in (the portable core) #
Copy these from the reference container or the distribution bundle. They contain no secrets and are identical across installs:
| File | Role |
|---|---|
bin/ssh-via-proxy | The SSH engine: stdio bridge for ssh -o ProxyCommand, carries SSH inside an HTTP CONNECT tunnel through the egress proxy. Pure Python, no deps. |
bin/gcp-tunnel-up.sh | Supervisor for the GCP reverse-SSH tunnel. flock-guarded singleton; does a stale-listener sweep on the GCP host before dialing; logs to workspace/tunnel/gcp-tunnel.log. Per-machine config at the top: REMOTE_FWD_PORT (SSH) and TERMINAL_FWD_PORT (terminal) — set both from the registry before first run. |
bin/recover-after-rebuild.sh | Idempotent rebuild recovery: fixes apt mirror, installs openssh-client/server + tmux, recreates the login user, restores authorized_keys from the persistent backup, starts supervisors. |
bin/fix-apt-mirror.sh | Drops the dead second Ubuntu mirror that makes apt-get update hang forever. Re-apply after every rebuild (/etc is ephemeral). |
bin/ttyd | ttyd 1.7.7 binary (browser terminal server). |
bin/ttyd-auth-proxy.py | HTTP Basic → signed session-cookie proxy in front of ttyd. Required because iOS Safari will not resend cached Basic credentials on the WebSocket /ws upgrade. Listens on 127.0.0.1:7681; ttyd itself runs with NO auth on 127.0.0.1:7682 (localhost-only). |
bin/ttyd-shell.sh | Per-connection shell wrapper: env TERM=xterm-256color tmux new-session -A -s main. Keep the volatile command here so ttyd restarts don't need supervisor restarts. |
bin/tunnel-up.sh | Legacy browser-tunnel supervisor (localhost.run). Only the original container uses it. Do NOT enable on new installs. |
Per-install items you must NOT copy: ~/.ssh/*, workspace/.ttyd-pass, workspace/.ttyd-cookie-secret, workspace/tunnel/muse-authorized_keys (the login key backup is per-install).
3. Provision the container #
Run bin/recover-after-rebuild.sh. It is idempotent and safe to re-run. What it does on a fresh container:
- Fixes the apt mirror (
fix-apt-mirror.sh). - Installs
openssh-client,openssh-server,tmux. - Creates the login user (currently
muse) — remember/home/muse
vanishes on rebuild, so the user is recreated here every time. 4. Restores authorized_keys from the persistent backup (workspace/.ssh-keys/ — keep your backups here; the script restores both the login user and root keys). 5. Sets PermitRootLogin prohibit-password in sshd_config (wiped on rebuild) and starts sshd on port 22. 6. Starts the tunnel supervisor(s) via flock-guarded ensure functions.
After provisioning, verify: ss -tln | grep :22 shows sshd listening.
4. The SSH engine — how dialing out works #
Every outbound SSH connection goes through ssh-via-proxy:
ssh -o ProxyCommand="$HOME/workspace/bin/ssh-via-proxy %h %p" \
-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
-o ServerAliveInterval=25 -o ServerAliveCountMax=3 \
-o ConnectTimeout=30 -o BatchMode=yes \
-N -R 127.0.0.1:<YOUR-PORT>:localhost:22 \
[email protected]
Key facts:
- The egress proxy password rotates per exec session. A long-lived
SSH connection keeps working (its CONNECT is already established), but any NEW dial must use the current session's env. This is why supervision lives in short-lived processes and crons, not in one eternal daemon. - The proxy kills long-lived CONNECT tunnels roughly every 10–15 minutes ("Broken pipe"). Expect redials; the supervisor handles them. - When our side dies, the remote sshd session can linger and keep the listen port bound, so redials fail with "remote port forwarding failed". gcp-tunnel-up.sh runs a pre-dial stale-listener sweep (fuser -k <port>/tcp on the GCP host, only when no local tunnel exists), and the GCP sshd has ClientAliveInterval 30 so ghosts are reaped in ~90s. If you see repeated "forwarding failed" errors, check the GCP side for lingering sshd-session processes.
5. Start the GCP supervisor #
bin/gcp-tunnel-up.sh (edit REMOTE_FWD_PORT, TERMINAL_FWD_PORT and SSH_KEY for this install first — take the ports from the registry, never invent them). Run it once in the background; it daemonizes and holds a flock on workspace/tunnel/gcp-tunnel.lock so double-starts exit cleanly. Verify: pgrep -f 'workspace/bin/gcp-tunnel-up\.sh$' and check workspace/tunnel/gcp-tunnel.log for "tunnel established".
The user reaches the container via the GCP VM: ssh -p <YOUR-PORT> <login-user>@localhost from the GCP VM (or ssh -J super@<gcp-ip> -p <YOUR-PORT> <login-user>@localhost from anywhere).
6. Browser terminal stack (same on every install) #
- Internal ttyd:
ttyd -p 7682 -i 127.0.0.1 -W bin/ttyd-shell.sh -Wis REQUIRED — without it the terminal is read-only and the page
still loads fine, so the failure is silent. - ttyd spawns its child command only after the client sends {"AuthToken": base64(user:pass), "columns": N, "rows": M} on the tty websocket subprotocol. An empty tmux ls with no client connected is EXPECTED, not a bug. - The shell wrapper exports TERM=xterm-256color; without it tmux dies with "open terminal failed". 2. Auth proxy on 127.0.0.1:7681 (ttyd-auth-proxy.py). Requires Basic once, issues a signed HttpOnly Secure session cookie, accepts cookie-or-Basic on /, /token, /ws. Cookie secret must be generated per install — never copy it. 3. tmux new-session -A -s main gives the persistent session that survives reloads and redials. 4. Generate workspace/.ttyd-pass (0600) for the muse login user.
7. Watchdog cron (runtime-side, survives rebuilds) #
VM-local processes die on rebuilds; the cron does not. Create a tunnel-watchdog cron, interval every 5 minutes, with the body from workspace/tunnel-duplicate/tunnel-watchdog.cron.md (in the bundle). Its job:
- Run
recover-after-rebuild.sh(idempotent — restarts supervisors
after a rebuild). 2. Read the current public URL from workspace/tunnel/URL.txt. 3. Probe it with a fresh environment (fresh proxy creds every run). HTTP 200 or 401 = healthy. 4. On 2 consecutive failures (~10 min): kill the tunnel's ssh PID(s) individually (exact PIDs via pgrep — NEVER a broad pkill), let the supervisor redial, reset the counter in workspace/tunnel/health.state.
Health-check split is deliberate: the supervisor does LOCAL supervision only (restart dead ttyd/proxy, redial when ssh exits). It never probes the public URL because its own env goes stale and its curls fail with 000 — false-positive redials, each minting a new public URL.
8. Verification checklist #
- [ ]
sshdials out through the proxy:ssh-via-proxygets a 200 CONNECT. - [ ]
gcp-tunnel-up.shrunning (pgrep),gcp-tunnel.logshows "tunnel established". - [ ] From the GCP VM:
ssh -p <YOUR-PORT> <login-user>@localhostlands in the container. - [ ] ttyd on 7682 + auth proxy on 7681 running; synthetic WebSocket
client with AuthToken JSON gets a shell (don't rely on curl). - [ ] tunnel-watchdog cron exists and its runs succeed. - [ ] Kill the ssh tunnel manually; supervisor redials within ~30s. - [ ] Simulate a rebuild mentally: everything you need is under /home/hatch; nothing in /etc, /home/muse, or process state.
9. Hard-won lessons (do not relearn these) #
- flock fds are inherited by daemonized children. Hold the lock only
in a subshell around check+start, and close the fd (9>&-) on the daemon's command line — otherwise the lock is held forever and every future run skips. - Anchor pgrep patterns (workspace/bin/tunnel-up\.sh$); unanchored patterns match sibling scripts (e.g. gcp-tunnel-up.sh). - Never broad-pkill in this environment. pkill -f "pattern" can match your own shell's command line and SIGTERM your session. Capture exact PIDs with pgrep (excluding $$) and kill those. - Only /home/hatch persists. Verified: everything else is on the ephemeral root overlay. Rebuilds happen without warning. - Apt mirror: stock ubuntu.sources lists a dead mirror that hangs apt-get update with no output. fix-apt-mirror.sh after every rebuild. - Container root cannot modify files owned by nobody (user-namespace quirk, EACCES). Delete and recreate instead of editing in place. - Dead ends, don't retry: Tailscale (proxy 400s the Noise handshake), cloudflared (ignores proxy env; API 403s quick tunnels from this egress IP), tmate (proxy blocks the whole tmate.io domain at destination level). The user-controlled GCP jump host is the viable path.
10. What stays per-install (never copy between containers) #
~/.ssh/* (the GCP keypair), workspace/.ttyd-pass, workspace/.ttyd-cookie-secret, login-user authorized_keys backups, the chosen REMOTE_FWD_PORT / TERMINAL_FWD_PORT, and the ttyd login password. Everything else in bin/ and this doc is identical across installs.
11. Onboarding a new machine (GCP VM side) #
The VM keeps a port registry and a helper script at /home/super/machine-registry/:
PORTS.md— every allocated machine, its SSH port, terminal port, and
subdomain. Read it before assigning ports. - add-machine.sh <name> <ssh-port> <terminal-port> — guards against bad names, taken ports, and duplicates; appends <name>.muse-dev.online → 127.0.0.1:<terminal-port> to the Caddyfile (validated before the live file is touched); reloads Caddy; records the allocation. Run with sudo (passwordless sudo is configured).
After the VM side is registered, the new machine's container dials in with its gcp-tunnel-up.sh set to the assigned ports — its terminal then appears at https://<name>.muse-dev.online with automatic HTTPS.