fix(remote): load a remote session's scrollback from the remote tmux

docs/architecture-invariants.md

@@ -77,7 +77,7 @@ OMP's registry entry declares `privilegedParams: []` and a launch spec that only
 
 Further detail: ⚠️ **Auto-reconnect revives ONLY when the durable remote tmux session is verifiably still alive** (`remoteTmuxSessionAlive()`, a `has-session` probe over ssh, #355): a clean agent exit (Ctrl-C, Ctrl-D, `exit`) tears that session down, and `isPaneDead()` cannot tell it from a transport drop, so the watcher used to relaunch a FRESH agent after every clean exit (claude only looked fine because its `|| --resume` fallback masked it). An unreachable host answers `undefined`, which also means do not revive. ⚠️ `has-session` prints NOTHING on success, so the probe is classified by EXIT STATUS (`classifyRemoteAliveExit`: 0 alive, ssh's 255 or a timeout unknown, anything else gone); reading stdout classified every live session as gone and silently disabled transport-drop reconnects. The answer is cached per session and forgotten whenever the pane is seen alive again, or a stale `true` from one transport drop would revive the next clean exit.
 
-File reads in a remote case are the second ssh surface (see Remote SSH cases below). The reason writes, office previews and thumbnails are deliberately unsupported over ssh is that no remote file is ever copied onto the server's disk. ssh children are bounded because terminal output in a remote session is written on the remote host, so a prompt-injected agent there can print hundreds of `codeman://attach` links.
+File reads in a remote case are the second ssh surface (see Remote SSH cases below). The reason writes, office previews and thumbnails are deliberately unsupported over ssh is that no remote file is ever copied onto the server's disk. ssh children are bounded because terminal output in a remote session is written on the remote host, so a prompt-injected agent there can print hundreds of `codeman://attach` links. The third surface is a remote session's full-history load, which reads the scrollback from the remote tmux (`src/remote-pane-history.ts`) through the same limiter, with a bounded slot wait (`maxWaitMs`) since the local capture is its fallback.
 
 ### Remote SSH cases
 

docs/remote-sessions.md

@@ -251,6 +251,34 @@ unreachable host answers "unknown", which also means do not revive. The answer
 is cached per session and cleared whenever the pane is next seen alive, so a
 stale `true` from one transport drop can never revive the NEXT clean exit.
 
+## Terminal history from the remote tmux
+
+The local pane runs the ssh attach, so it only sees the remote tmux client's
+full-screen repaints and keeps a fraction of the scrollback (none after a
+reattach). A full-history load (`GET /api/sessions/:id/terminal?full=1`) of a
+remote session therefore reads the scrollback rows from the REMOTE pane in one
+ssh round trip (`src/remote-pane-history.ts`) and splices them above the local
+visible frame and caret. Bounded (5 s for the round trip, at most 1.5 s queued
+for a slot in the shared ssh limiter) and fail-soft: any failure keeps the
+local capture, and a transport failure (ssh exit 255, a timeout, output that is
+not the script's) also skips that host for 30 s; a failure of the remote
+command itself (a remote tmux session that is gone) affects that session only.
+How much history exists is the remote tmux's own `history-limit`, which the
+launch command does not set (tmux's default is 2000 lines).
+
+Two seams can drop rows at the join, silently. Neither loses anything that
+loaded before this existed:
+
+- **Rows that scroll during the round trip.** The remote history is read
+  first and the local frame after it, so rows that scrolled off the remote
+  screen in between are in neither (measured: 13 lines lost with a 150 ms
+  round trip and fast output).
+- **A shared remote session with a taller client.** Under `window-size latest`
+  a remote session attached elsewhere is sized by its latest client; when that
+  client is taller than this pane, the rows between this pane's frame and the
+  remote screen's top are in neither the remote history nor the local frame
+  (measured: 20 rows).
+
 ## File access over SSH
 
 A remote case's `workingDir` is an absolute path on the **remote** host

src/mux-interface.ts

@@ -189,6 +189,13 @@ export interface PaneCaptureOptions {
   fullHistory?: boolean;
   /** Bound the full-history capture to this many scrollback lines (`-S -<N>`). */
   historyLimitLines?: number;
+  /**
+   * Full-history only: scrollback rows read from somewhere other than this pane
+   * (raw `capture-pane -p -e -J` text, oldest first). The pane then contributes
+   * only its visible frame and caret, and its own history is not read. A remote
+   * session's real history lives in the REMOTE tmux (`remote-pane-history.ts`).
+   */
+  scrollbackOverride?: string;
   /**
    * Byte cap the consumer will keep from the capture. Sizes the child-process
    * stdout buffer (with slack) so multi-MB scrollback dumps aren't killed by

src/remote-pane-history.ts

@@ -0,0 +1,210 @@
+/**
+ * @fileoverview Scrollback of a remote SSH session, read from the REMOTE tmux.
+ *
+ * A remote session's local pane runs `ssh -t … tmux new-session -A` (or `attach`),
+ * so the local tmux only ever sees what the remote tmux CLIENT draws: full-screen
+ * repaints, not scrolling output. The local pane therefore keeps a fraction of the
+ * history (measured: 48 of 279 lines while output streamed, none after a reattach),
+ * while the remote pane, where the shell really runs, keeps all of it. A full-history
+ * load of a remote session reads the scrollback from there instead, and the caller
+ * splices it above the LOCAL visible frame (`PaneCaptureOptions.scrollbackOverride`):
+ * the local pane is what the live stream keeps drawing into, so its frame and caret
+ * position are the ones the browser must end up with.
+ *
+ * ONE ssh round trip prints the remote pane's `#{history_size}` and its scrollback
+ * rows (`capture-pane -S -<N> -E -1`). The size is load-bearing: with no history at
+ * all, tmux clamps `-E -1` to the first VISIBLE row and would return it as history.
+ *
+ * Bounded and fail-soft. Any failure (unreachable host, auth, missing session, odd
+ * output) returns null and the caller keeps the local capture, so a remote load is
+ * never worse than before. A transport failure also starts a short per-host back-off,
+ * so a host that is asleep costs one timeout, not one per tab switch. The child runs
+ * under the shared remote-ssh limiter (`remote-ssh-limiter.ts`) like every other
+ * short-lived remote ssh, but gives up on a slot it waited too long for (`maxWaitMs`,
+ * enforced in the queue), and the connection options come from
+ * `buildSshConnectionArgs()`.
+ *
+ * How much history exists is the REMOTE tmux's `history-limit` (2000 lines unless
+ * the remote user's tmux.conf raises it); the launch command does not set it.
+ */
+
+import { exec } from 'node:child_process';
+import { promisify } from 'node:util';
+import type { SessionRemote } from './types/session.js';
+import { buildSshConnectionArgs, classifyRemoteAliveExit, remoteSshTarget, shellescape } from './remote-hosts.js';
+import { RemoteSshSlotTimeoutError, runWithRemoteSshLimit } from './remote-ssh-limiter.js';
+import { remoteTmuxLocation } from './tmux-manager.js';
+
+const execAsync = promisify(exec);
+
+/**
+ * Bound on the whole round trip. A full load waits on it, so it is far below the
+ * 10 s `ConnectTimeout` the shared ssh options carry: past this the local capture
+ * is the better answer.
+ */
+const REMOTE_HISTORY_TIMEOUT_MS = 5_000;
+
+/**
+ * Longest a load waits for a slot in the shared ssh pool, which file previews and
+ * attachment probes (20 s each) also hold. Past it the local capture is used.
+ */
+const REMOTE_HISTORY_SLOT_WAIT_MS = 1_500;
+
+/** After a failed read, skip that host for this long and use the local capture. */
+const REMOTE_HISTORY_BACKOFF_MS = 30_000;
+
+/** Headroom over the remote `tail -c` bound for the size line and markers. */
+const REMOTE_HISTORY_SLACK_BYTES = 64 * 1024;
+
+/** Read ceiling when the configured byte cap is 0 (unbounded) and no `tail -c` runs. */
+const UNBOUNDED_READ_MAX_BYTES = 64 * 1024 * 1024;
+
+/**
+ * Host id → time the back-off ends. Keyed by HOST, so a sleeping host with five
+ * sessions costs one timeout, and the map is bounded by the configured hosts.
+ */
+const backoffUntil = new Map<string, number>();
+
+/**
+ * The remote shell script. Its first byte is a NUL so a login banner or an rc-file
+ * `echo` printed before it runs is separated without guessing (the same marker
+ * `buildRemoteProbeCommand` uses); a second NUL ends the history size.
+ */
+export function buildRemoteScrollbackScript(
+  socket: string,
+  sessionName: string,
+  historyLines: number,
+  maxBytes: number
+): string {
+  const tmux = `tmux -L ${shellescape(socket)}`;
+  // `=name:` is an EXACT session match, then its active pane. A bare name is
+  // read as a pane index or window name first, so a discovered session named
+  // `0` would capture some other session's pane.
+  const target = shellescape(`=${sessionName}:`);
+  const lines = Math.max(1, Math.trunc(historyLines));
+  // The newest rows are the ones worth keeping, so the byte bound is applied on
+  // the remote side (`tail -c`) rather than by killing the read at maxBuffer.
+  // A cap of 0 means unbounded, as in the route.
+  const bound = maxBytes > 0 ? ` | tail -c ${Math.trunc(maxBytes)}` : '';
+  return (
+    `printf '\\0' && ${tmux} display-message -p -t ${target} '#{history_size}' && printf '\\0' && ` +
+    `${tmux} capture-pane -p -e -J -S -${lines} -E -1 -t ${target}${bound}`
+  );
+}
+
+/** The full ssh command line for {@link buildRemoteScrollbackScript}. */
+export function buildRemoteScrollbackCommand(
+  remote: SessionRemote,
+  sessionId: string,
+  historyLines: number,
+  maxBytes: number
+): string {
+  const { socket, sessionName } = remoteTmuxLocation(remote, sessionId);
+  const script = buildRemoteScrollbackScript(socket, sessionName, historyLines, maxBytes);
+  // `exec` so a timeout kill reaches ssh itself, not only the local `sh -c`.
+  return ['exec', ...buildSshConnectionArgs(remote), remoteSshTarget(remote), shellescape(script)].join(' ');
+}
+
+/**
+ * A `tail -c` cut lands mid-row (possibly mid-escape) whenever it bit, so the
+ * first, partial row is dropped. Under the bound, the rows are whole.
+ */
+export function dropPartialFirstRow(scrollback: string, maxBytes: number): string {
+  if (maxBytes <= 0 || Buffer.byteLength(scrollback, 'utf-8') < maxBytes) return scrollback;
+  const newline = scrollback.indexOf('\n');
+  return newline === -1 ? '' : scrollback.slice(newline + 1);
+}
+
+/**
+ * The scrollback rows from the script's stdout, '' when the pane has no history,
+ * or null when the output is not the script's.
+ */
+export function parseRemoteScrollbackOutput(stdout: string): string | null {
+  const start = stdout.indexOf('\0');
+  if (start === -1) return null;
+  const sizeEnd = stdout.indexOf('\0', start + 1);
+  if (sizeEnd === -1) return null;
+  const sizeText = stdout.slice(start + 1, sizeEnd).trim();
+  if (!/^\d+$/.test(sizeText)) return null;
+  if (Number(sizeText) === 0) return '';
+  return stdout.slice(sizeEnd + 1);
+}
+
+/** Runs the ssh command line and resolves its stdout; rejects like `exec` does. */
+export type RemoteScrollbackRunner = (command: string, opts: { timeout: number; maxBuffer: number }) => Promise<string>;
+
+const execRunner: RemoteScrollbackRunner = async (command, opts) =>
+  (await execAsync(command, { encoding: 'utf-8', ...opts })).stdout;
+
+/**
+ * Read a remote session's scrollback from the remote tmux, or null to keep the
+ * local capture. Never throws.
+ *
+ * Only a TRANSPORT failure backs the host off (ssh's own exit 255, a timeout, a
+ * spawn error, output that is not the script's): those say the host is
+ * unreachable for every session on it. A remote command failing (a session
+ * that is gone: tmux exits 1, or answers nothing between the markers when the
+ * `| tail -c` pipe hides its status) is about this session only, so it returns
+ * null and the next session on that host still tries. Same split as
+ * `classifyRemoteAliveExit()`.
+ *
+ * `run` is a test seam: without it nothing runs under vitest, as in
+ * remote-files.ts / remote-hosts.ts.
+ */
+export async function fetchRemoteScrollback(
+  remote: SessionRemote,
+  sessionId: string,
+  historyLines: number,
+  maxBytes: number,
+  run?: RemoteScrollbackRunner
+): Promise<string | null> {
+  if (!run && process.env.VITEST) return null;
+  const runner = run ?? execRunner;
+  const until = backoffUntil.get(remote.hostId);
+  if (until !== undefined) {
+    if (Date.now() < until) return null;
+    backoffUntil.delete(remote.hostId);
+  }
+  const command = buildRemoteScrollbackCommand(remote, sessionId, historyLines, maxBytes);
+  try {
+    const stdout = await runWithRemoteSshLimit(
+      () =>
+        runner(command, {
+          timeout: REMOTE_HISTORY_TIMEOUT_MS,
+          maxBuffer: (maxBytes > 0 ? maxBytes : UNBOUNDED_READ_MAX_BYTES) + REMOTE_HISTORY_SLACK_BYTES,
+        }),
+      // The pool is shared with 20 s file probes; past this the local capture
+      // is the better answer than waiting behind them.
+      { maxWaitMs: REMOTE_HISTORY_SLOT_WAIT_MS }
+    );
+    const scrollback = parseRemoteScrollbackOutput(stdout);
+    if (scrollback === null) {
+      // Both markers arrived, so the script ran on the host and only tmux had
+      // nothing to say: no such session. The `| tail -c` pipe hides tmux's
+      // exit status, so this comes back as exit 0 rather than exit 1, and it
+      // is about this session alone. Output without the markers is not the
+      // script's at all, which says the host is not answering as expected.
+      const ranOnHost = stdout.split('\0').length >= 3;
+      if (!ranOnHost) backoffUntil.set(remote.hostId, Date.now() + REMOTE_HISTORY_BACKOFF_MS);
+      const what = ranOnHost ? 'no such remote tmux session' : 'unexpected output';
+      console.warn(`[remote-history] ${remote.label}: ${what}; using the local capture`);
+      return null;
+    }
+    return dropPartialFirstRow(scrollback, maxBytes);
+  } catch (err) {
+    // A busy pool says nothing about the host.
+    if (err instanceof RemoteSshSlotTimeoutError) return null;
+    const e = err as { killed?: boolean; code?: unknown };
+    const code = typeof e.code === 'number' ? e.code : null;
+    const transport = classifyRemoteAliveExit(code, e.killed === true) === undefined;
+    if (transport) backoffUntil.set(remote.hostId, Date.now() + REMOTE_HISTORY_BACKOFF_MS);
+    const reason = e.killed ? 'timed out' : code !== null ? `exit ${code}` : 'failed';
+    console.warn(`[remote-history] ${remote.label}: remote scrollback read ${reason}; using the local capture`);
+    return null;
+  }
+}
+
+/** Test hook: forget every host back-off. */
+export function resetRemoteScrollbackBackoff(): void {
+  backoffUntil.clear();
+}

src/remote-ssh-limiter.ts

@@ -1,7 +1,9 @@
 /**
  * @fileoverview Global concurrency limiter for the short-lived `ssh` children that
  * remote-case file access spawns (`src/remote-files.ts`: the realpath+stat probe and
- * the buffered text read).
+ * the buffered text read). A remote session's full-history load reads its scrollback
+ * from the remote tmux through the same pool (`src/remote-pane-history.ts`), with a
+ * bounded wait (`maxWaitMs`) because it has the local capture to fall back on.
  *
  * Two paths can fan those out without a human behind each one:
  *
@@ -60,12 +62,35 @@ export function getRemoteSshLimit(): number {
   return MAX_CONCURRENT_REMOTE_SSH;
 }
 
-function acquire(): Promise<void> {
+/** Thrown by {@link runWithRemoteSshLimit} when `maxWaitMs` passed before a slot freed up. */
+export class RemoteSshSlotTimeoutError extends Error {
+  constructor(maxWaitMs: number) {
+    super(`No remote ssh slot within ${maxWaitMs} ms`);
+    this.name = 'RemoteSshSlotTimeoutError';
+  }
+}
+
+function acquire(maxWaitMs?: number): Promise<void> {
   if (active < MAX_CONCURRENT_REMOTE_SSH) {
     active++;
     return Promise.resolve();
   }
-  return new Promise<void>((resolve) => waiters.push(resolve));
+  if (maxWaitMs === undefined) return new Promise<void>((resolve) => waiters.push(resolve));
+  return new Promise<void>((resolve, reject) => {
+    const waiter = () => {
+      clearTimeout(timer);
+      resolve();
+    };
+    // A timed-out waiter leaves the queue and is never handed a slot, so it
+    // owns nothing to give back: `active` is untouched on this path.
+    const timer = setTimeout(() => {
+      const index = waiters.indexOf(waiter);
+      if (index === -1) return; // already handed a slot by release()
+      waiters.splice(index, 1);
+      reject(new RemoteSshSlotTimeoutError(maxWaitMs));
+    }, maxWaitMs);
+    waiters.push(waiter);
+  });
 }
 
 function release(): void {
@@ -78,9 +103,14 @@ function release(): void {
   }
 }
 
-/** Run `task` once an ssh slot is free, releasing the slot afterward. */
-export async function runWithRemoteSshLimit<T>(task: () => Promise<T>): Promise<T> {
-  await acquire();
+/**
+ * Run `task` once an ssh slot is free, releasing the slot afterward. With
+ * `maxWaitMs`, give up instead (rejecting with {@link RemoteSshSlotTimeoutError},
+ * `task` never runs) when no slot frees up in time: for a caller that has a
+ * cheaper answer than waiting behind the queue.
+ */
+export async function runWithRemoteSshLimit<T>(task: () => Promise<T>, opts: { maxWaitMs?: number } = {}): Promise<T> {
+  await acquire(opts.maxWaitMs);
   try {
     return await task();
   } finally {