Muse Front Door — docs

Outbound terminal tunnel

Public browser terminal into this container, no inbound ports needed. Stable URL (never changes): https://muse-dev.online

How it works


iPhone browser --HTTPS--> muse-dev.online (Caddy on GCP VM, auto Let's Encrypt)

    --reverse_proxy--> GCP 127.0.0.1:7681

    --SSH -R 127.0.0.1:7681--> container ssh --CONNECT--> egress proxy --> GCP VM 34.139.37.135:22

    --> ttyd-auth-proxy 127.0.0.1:7681 (Basic once -> signed session cookie)

    --> ttyd 127.0.0.1:7682 (NO auth, localhost-only, -W writable)

    --> ~/workspace/bin/ttyd-shell.sh --> tmux new-session -A -s main

The same SSH session also forwards GCP 127.0.0.1:2224 → container localhost:22 for native SSH access (ssh -J [email protected] -p 2224 muse@localhost). Both forwards are supervised by ~/workspace/bin/gcp-tunnel-up.sh, which redials on drop — the public URL survives redials because it is tied to the GCP VM's static IP, not to any single SSH session.

Why the proxy: iOS Safari sends cached HTTP Basic credentials for the page load but NOT on the WebSocket /ws upgrade (or on fetch() to /token), so a Basic-auth-only ttyd left iPhones stuck in a reconnect loop (server log: User code denied connection). The proxy requires Basic once, then issues a signed HttpOnly; Secure session cookie; /, /token, and /ws accept the cookie OR Basic. Safari always sends cookies — including on WS upgrades — so the handshake now succeeds. Internal ttyd has no auth of its own and must never be exposed directly; only the proxy's port is tunnel-forwarded. (~/workspace/bin/ttyd-auth-proxy.py, secret in ~/.ttyd-cookie-secret.)

HTTP CONNECT tunnel through the egress proxy (pure python, no socat quirks). - ~/workspace/bin/gcp-tunnel-up.sh — supervisor: keeps the GCP reverse-SSH alive (-R 127.0.0.1:2224:localhost:22 for SSH, -R 127.0.0.1:7681:localhost:7681 for the terminal), redials on drop, sweeps stale GCP-side listeners before redialing. The public URL is stable because it lives on the GCP VM, not on the SSH session. - (RETIRED 2026-10-02) ~/workspace/bin/tunnel-up.sh — the old localhost.run supervisor. No longer started; recover-after-rebuild.sh keeps it down. Kept in the repo only as a fallback. - ~/workspace/bin/ttyd-shell.sh — the command ttyd runs per connection, kept as a separate file so it can change without restarting the supervisor. It attaches to (or creates) tmux session main, so a reloaded page or a tunnel redial lands back in the same shell with scrollback intact. - ttyd creds: user muse, password in ~/.ttyd-pass (0600). The password is checked by the auth proxy, not by ttyd itself (internal ttyd runs without -c, so the password no longer appears in any process list). - ttyd needs -W: without it the terminal is readonly (user can't type). Also note: ttyd spawns its child only after the browser client sends {"AuthToken": base64(user:pass), ...} on the tty websocket subprotocol — no client, no child process, so tmux ls is empty until first connect. - URL is stable: https://muse-dev.online is served by Caddy on the GCP VM (static IP 34.139.37.135, automatic Let's Encrypt certificate) and never changes, even across SSH redials or container rebuilds.

Health checking (split by design)

Public-URL health probing and local supervision are deliberately separated:

redials when it drops (with a stale-listener sweep on the GCP host first). ttyd (7682) and the auth proxy (7681) are supervised alongside it. These checks touch only localhost, so they never need the egress proxy. - Public — the tunnel-watchdog runtime cron (every 5 min) probes the stable URL https://muse-dev.online/. This must live in the cron, not the supervisor: the egress proxy password rotates per exec session, so the long-lived supervisor's curls go stale and fail with 000 — probing there caused false-positive redials. Each cron run gets a fresh environment, so its verdicts are trustworthy. Two consecutive failed probes (~10 min) → the cron kills the GCP ssh tunnel PID(s) and the supervisor's main loop redials; the same public URL comes back. A single 000 from the supervisor's own post-dial check is treated as inconclusive, never as failure; only a definite edge error (5xx) redials immediately. - Consecutive-failure state lives in ~/workspace/tunnel/health.state (integer, reset to 0 on any healthy probe or when the ssh session is mid-redial).

Operate

pgrep -f "[t]tyd -p 7682" - On the GCP VM: sudo systemctl status caddy, cat /etc/caddy/Caddyfile (muse-dev.online → reverse_proxy 127.0.0.1:7681)

GCP jump-host relay (native SSH, no browser)

The same reverse-SSH session that carries the browser terminal also forwards GCP 127.0.0.1:2224 → container localhost:22 (container sshd, login user muse): a reverse-SSH from this container to the user's GCP VM (34.139.37.135, user super, key ~/.ssh/vm_to_gcp). Connect with ssh -J [email protected] -p 2224 muse@localhost (or from the GCP VM: ssh -p 2224 muse@localhost). This is the intended shared front door — a second container dials the same box on its own ports.

~/workspace/tunnel/gcp-tunnel.log). Same 10–15 min proxy severing as before; before each (re)dial it sweeps stale remote listeners (fuser -k 2224/tcp 7681/tcp) because dead GCP-side sessions linger and would otherwise block the rebind ("remote port forwarding failed"). GCP sshd has ClientAliveInterval 30 so ghosts are reaped in ~90 s as a backstop. - The tunnel-watchdog cron covers it too: recover-after-rebuild.sh ensures the supervisor (own lock, anchored pgrep) and installs openssh-server on fresh containers.

Rebuild recovery (automated)

The tunnel-watchdog runtime cron (every 5 min) runs ~/workspace/bin/recover-after-rebuild.sh, which is idempotent: on a fresh container it re-provisions (apt mirror fix, openssh-client, python3, tmux) and restarts the supervisor; on a healthy container it's a no-op. Crons survive container rebuilds; VM-local processes don't. ttyd binary, ssh-via-proxy, and all scripts live under ~/workspace (persistent).

Full runbook: RECOVERY.md — one-command recovery via ~/workspace/bin/recover-after-rebuild.sh (validated 2026-10-01).

Verified 2026-10-01

End-to-end through the public URL: unauthenticated -> 401, basic-auth -> 200 + session cookie, cookie-only /token -> 200, cookie-only WebSocket /ws upgrade (no Authorization header, exactly what iOS Safari sends) -> 101, shell I/O round-trips (COOKIE_TTY_OK 42), unauthenticated /ws -> 401. Self-healing is split by design (see "Health checking" above): the supervisor keeps the SSH session alive and redials when it exits, while the tunnel-watchdog cron owns public-URL probing with fresh proxy credentials — 2 consecutive failed probes (~10 min) kills the ssh session and forces a redial. Probing inside the supervisor was removed 2026-10-01 after discovering the egress proxy password rotates: the supervisor's stale curls failed with 000 and caused false-positive redials. (2026-10-02: the localhost.run edge — and its URL churn — was retired entirely in favor of the stable GCP+Caddy URL.)

Full terminal chain verified with synthetic WS clients (stdlib python): ~/workspace/tunnel/ws-selftest.py (Basic-auth flow) and ~/workspace/tunnel/ws-cookie-test.py (the Safari flow: login once, then cookie-only for /token and /ws). Handshake on the tty subprotocol + {"AuthToken": ...} -> ttyd spawns ttyd-shell.sh -> tmux session main created, terminal output received, session persists after disconnect. tmux needs TERM set (the wrapper exports TERM=xterm-256color); without it tmux dies with "terminal does not support clear".