Muse Front Door — docs
Contents

INSTALL.md — R&D container networking setup (agent-facing)

Scope: replicate this container's SSH and terminal R&D stack on another, identical container. The container image is the same everywhere; only the per-install identity (keys, ports, secrets) changes.

The architecture, in one picture:


  ┌─ identical container ──────────────────────────────┐

  │  ssh-via-proxy (CONNECT relay through egress proxy) │

  │  gcp-tunnel-up.sh ── ssh -R 127.0.0.1:<PORT>:localhost:22

  │  ttyd 1.7.7 + ttyd-auth-proxy.py + tmux ("main")    │

  │  tunnel-watchdog cron (every 5 min, runtime-side)   │

  └─────────────────────────┬──────────────────────────┘

                            │  dials OUT through the

                            │  egress proxy (rotating creds)

                            ▼

                    shared GCP jump host (34.139.37.135)

                    user reaches the container via GCP

One rule that governs everything below: the GCP VM is the shared front door. New containers dial INTO it. There are no per-container localhost.run browser tunnels — that path exists only on the original container and is legacy. The GCP side will reverse-proxy browser traffic too; the container-side browser stack (ttyd, auth proxy, tmux) is the same regardless of which public path fronts it.


1. Before you start (prerequisites) #

You need from the human (or from the GCP owner):

  1. GCP VM address and SSH user — currently 34.139.37.135, user super.
  2. Unique ports on the GCP VM for THIS container — check the registry

first: /home/super/machine-registry/PORTS.md on the GCP VM lists every taken port. You need one SSH port and one terminal port, both free (this container owns 2224/7681; the operator uplink super owns 2222). Two machines sharing a port will flap forever ("remote port forwarding failed"). 3. A fresh SSH keypair generated inside this container (ssh-keygen -t ed25519 -f ~/.ssh/vm_to_gcp -N ""). The PUBLIC key must be added to the GCP VM's super user authorized_keys. The private key never leaves the container. 4. Egress proxy env vars present (HTTPS_PROXY/https_proxy pointing at hatch-egress-proxy:3128). Verify: env | grep -i proxy. Without these, nothing below works. 5. A ttyd login password you generate yourself and hand to the user (user muse). Generate with openssl rand -hex 16.

Also confirm the container basics: /home/hatch is the only directory that survives a rebuild (/dev/mapper/rv, btrfs). /home/muse, /etc, apt packages, and all processes are wiped without warning. Everything durable must live under /home/hatch.

2. Files to bring in (the portable core) #

Copy these from the reference container or the distribution bundle. They contain no secrets and are identical across installs:

FileRole
bin/ssh-via-proxyThe SSH engine: stdio bridge for ssh -o ProxyCommand, carries SSH inside an HTTP CONNECT tunnel through the egress proxy. Pure Python, no deps.
bin/gcp-tunnel-up.shSupervisor for the GCP reverse-SSH tunnel. flock-guarded singleton; does a stale-listener sweep on the GCP host before dialing; logs to workspace/tunnel/gcp-tunnel.log. Per-machine config at the top: REMOTE_FWD_PORT (SSH) and TERMINAL_FWD_PORT (terminal) — set both from the registry before first run.
bin/recover-after-rebuild.shIdempotent rebuild recovery: fixes apt mirror, installs openssh-client/server + tmux, recreates the login user, restores authorized_keys from the persistent backup, starts supervisors.
bin/fix-apt-mirror.shDrops the dead second Ubuntu mirror that makes apt-get update hang forever. Re-apply after every rebuild (/etc is ephemeral).
bin/ttydttyd 1.7.7 binary (browser terminal server).
bin/ttyd-auth-proxy.pyHTTP Basic → signed session-cookie proxy in front of ttyd. Required because iOS Safari will not resend cached Basic credentials on the WebSocket /ws upgrade. Listens on 127.0.0.1:7681; ttyd itself runs with NO auth on 127.0.0.1:7682 (localhost-only).
bin/ttyd-shell.shPer-connection shell wrapper: env TERM=xterm-256color tmux new-session -A -s main. Keep the volatile command here so ttyd restarts don't need supervisor restarts.
bin/tunnel-up.shLegacy browser-tunnel supervisor (localhost.run). Only the original container uses it. Do NOT enable on new installs.

Per-install items you must NOT copy: ~/.ssh/*, workspace/.ttyd-pass, workspace/.ttyd-cookie-secret, workspace/tunnel/muse-authorized_keys (the login key backup is per-install).

3. Provision the container #

Run bin/recover-after-rebuild.sh. It is idempotent and safe to re-run. What it does on a fresh container:

  1. Fixes the apt mirror (fix-apt-mirror.sh).
  2. Installs openssh-client, openssh-server, tmux.
  3. Creates the login user (currently muse) — remember /home/muse

vanishes on rebuild, so the user is recreated here every time. 4. Restores authorized_keys from the persistent backup (workspace/.ssh-keys/ — keep your backups here; the script restores both the login user and root keys). 5. Sets PermitRootLogin prohibit-password in sshd_config (wiped on rebuild) and starts sshd on port 22. 6. Starts the tunnel supervisor(s) via flock-guarded ensure functions.

After provisioning, verify: ss -tln | grep :22 shows sshd listening.

4. The SSH engine — how dialing out works #

Every outbound SSH connection goes through ssh-via-proxy:


ssh -o ProxyCommand="$HOME/workspace/bin/ssh-via-proxy %h %p" \

    -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \

    -o ServerAliveInterval=25 -o ServerAliveCountMax=3 \

    -o ConnectTimeout=30 -o BatchMode=yes \

    -N -R 127.0.0.1:<YOUR-PORT>:localhost:22 \

    [email protected]

Key facts:

SSH connection keeps working (its CONNECT is already established), but any NEW dial must use the current session's env. This is why supervision lives in short-lived processes and crons, not in one eternal daemon. - The proxy kills long-lived CONNECT tunnels roughly every 10–15 minutes ("Broken pipe"). Expect redials; the supervisor handles them. - When our side dies, the remote sshd session can linger and keep the listen port bound, so redials fail with "remote port forwarding failed". gcp-tunnel-up.sh runs a pre-dial stale-listener sweep (fuser -k <port>/tcp on the GCP host, only when no local tunnel exists), and the GCP sshd has ClientAliveInterval 30 so ghosts are reaped in ~90s. If you see repeated "forwarding failed" errors, check the GCP side for lingering sshd-session processes.

5. Start the GCP supervisor #

bin/gcp-tunnel-up.sh (edit REMOTE_FWD_PORT, TERMINAL_FWD_PORT and SSH_KEY for this install first — take the ports from the registry, never invent them). Run it once in the background; it daemonizes and holds a flock on workspace/tunnel/gcp-tunnel.lock so double-starts exit cleanly. Verify: pgrep -f 'workspace/bin/gcp-tunnel-up\.sh$' and check workspace/tunnel/gcp-tunnel.log for "tunnel established".

The user reaches the container via the GCP VM: ssh -p <YOUR-PORT> <login-user>@localhost from the GCP VM (or ssh -J super@<gcp-ip> -p <YOUR-PORT> <login-user>@localhost from anywhere).

6. Browser terminal stack (same on every install) #

  1. Internal ttyd: ttyd -p 7682 -i 127.0.0.1 -W bin/ttyd-shell.sh
  2. -W is REQUIRED — without it the terminal is read-only and the page

still loads fine, so the failure is silent. - ttyd spawns its child command only after the client sends {"AuthToken": base64(user:pass), "columns": N, "rows": M} on the tty websocket subprotocol. An empty tmux ls with no client connected is EXPECTED, not a bug. - The shell wrapper exports TERM=xterm-256color; without it tmux dies with "open terminal failed". 2. Auth proxy on 127.0.0.1:7681 (ttyd-auth-proxy.py). Requires Basic once, issues a signed HttpOnly Secure session cookie, accepts cookie-or-Basic on /, /token, /ws. Cookie secret must be generated per install — never copy it. 3. tmux new-session -A -s main gives the persistent session that survives reloads and redials. 4. Generate workspace/.ttyd-pass (0600) for the muse login user.

7. Watchdog cron (runtime-side, survives rebuilds) #

VM-local processes die on rebuilds; the cron does not. Create a tunnel-watchdog cron, interval every 5 minutes, with the body from workspace/tunnel-duplicate/tunnel-watchdog.cron.md (in the bundle). Its job:

  1. Run recover-after-rebuild.sh (idempotent — restarts supervisors

after a rebuild). 2. Read the current public URL from workspace/tunnel/URL.txt. 3. Probe it with a fresh environment (fresh proxy creds every run). HTTP 200 or 401 = healthy. 4. On 2 consecutive failures (~10 min): kill the tunnel's ssh PID(s) individually (exact PIDs via pgrep — NEVER a broad pkill), let the supervisor redial, reset the counter in workspace/tunnel/health.state.

Health-check split is deliberate: the supervisor does LOCAL supervision only (restart dead ttyd/proxy, redial when ssh exits). It never probes the public URL because its own env goes stale and its curls fail with 000 — false-positive redials, each minting a new public URL.

8. Verification checklist #

client with AuthToken JSON gets a shell (don't rely on curl). - [ ] tunnel-watchdog cron exists and its runs succeed. - [ ] Kill the ssh tunnel manually; supervisor redials within ~30s. - [ ] Simulate a rebuild mentally: everything you need is under /home/hatch; nothing in /etc, /home/muse, or process state.

9. Hard-won lessons (do not relearn these) #

in a subshell around check+start, and close the fd (9>&-) on the daemon's command line — otherwise the lock is held forever and every future run skips. - Anchor pgrep patterns (workspace/bin/tunnel-up\.sh$); unanchored patterns match sibling scripts (e.g. gcp-tunnel-up.sh). - Never broad-pkill in this environment. pkill -f "pattern" can match your own shell's command line and SIGTERM your session. Capture exact PIDs with pgrep (excluding $$) and kill those. - Only /home/hatch persists. Verified: everything else is on the ephemeral root overlay. Rebuilds happen without warning. - Apt mirror: stock ubuntu.sources lists a dead mirror that hangs apt-get update with no output. fix-apt-mirror.sh after every rebuild. - Container root cannot modify files owned by nobody (user-namespace quirk, EACCES). Delete and recreate instead of editing in place. - Dead ends, don't retry: Tailscale (proxy 400s the Noise handshake), cloudflared (ignores proxy env; API 403s quick tunnels from this egress IP), tmate (proxy blocks the whole tmate.io domain at destination level). The user-controlled GCP jump host is the viable path.

10. What stays per-install (never copy between containers) #

~/.ssh/* (the GCP keypair), workspace/.ttyd-pass, workspace/.ttyd-cookie-secret, login-user authorized_keys backups, the chosen REMOTE_FWD_PORT / TERMINAL_FWD_PORT, and the ttyd login password. Everything else in bin/ and this doc is identical across installs.

11. Onboarding a new machine (GCP VM side) #

The VM keeps a port registry and a helper script at /home/super/machine-registry/:

subdomain. Read it before assigning ports. - add-machine.sh <name> <ssh-port> <terminal-port> — guards against bad names, taken ports, and duplicates; appends <name>.muse-dev.online → 127.0.0.1:<terminal-port> to the Caddyfile (validated before the live file is touched); reloads Caddy; records the allocation. Run with sudo (passwordless sudo is configured).

After the VM side is registered, the new machine's container dials in with its gcp-tunnel-up.sh set to the assigned ports — its terminal then appears at https://<name>.muse-dev.online with automatic HTTPS.