SIGSEGV in _tmxr_activate_delay after RESTORE when a telnet re-attach fails (UNIT_TM_POLL is restored, uptr->tmxr is not)

[ChristineTham]

SIGSEGV in _tmxr_activate_delay after RESTORE when a telnet re-attach fails (UNIT_TM_POLL is restored, uptr->tmxr is not)

---

Summary

SAVE persists a unit's dynflags, including UNIT_TM_POLL, but uptr->tmxr is a runtime pointer that only a successful tmxr_attach can set. If a RESTORE's recorded telnet re-attach fails, the unit comes back tagged as a mux polling unit with a NULL tmxr backpointer, and the next GO/CONTINUE dereferences NULL in _tmxr_activate_delay.

Restore reports success, so nothing signals that the machine is now one command away from a crash.

Reproduced on a stock vax780 with no disk image and no operating system — the guest is a two-instruction counter loop.

Version

VAX 11/780 simulator Open SIMH V4.1-0 Current    git commit id: a1f57fa3
Compiler: GCC Apple LLVM 21.0.0 (clang-2100.1.1.101)
Build Tool: simh-makefile (Release Build)
Host: macOS (Darwin 25.6.0), arm64

Asynchronous I/O support is compiled in (not enabled at run time — set asynch was never issued).

Repro

guest.ini — deposits INCL @#400 ; BRB .-8 at 0x200, attaches the console to telnet, saves. No telnet client ever needs to connect; the attach alone sets the flag.

set console telnet=7654
d -b 200 D6
d -b 201 9F
d -b 202 00
d -b 203 04
d -b 204 00
d -b 205 00
d -b 206 11
d -b 207 F8
dep pc 200
save crash.sav
exit

restore.ini:

restore crash.sav
go 200

./BIN/vax780 guest.ini                     # writes crash.sav, exits cleanly

# Occupy the console port so the restore's re-attach fails:
python3 -c "import socket,time; s=socket.socket();
s.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1);
s.bind(('127.0.0.1',7654)); s.listen(1); time.sleep(60)" &

./BIN/vax780 restore.ini                   # SIGSEGV

Exit status 11 (SIGSEGV). restore itself does not report failure here; a show console after the failed restore reports Connected to console window, i.e. the telnet console is gone while the TTI unit still carries UNIT_TM_POLL from the save file.

Crash

stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
  * frame #0: _tmxr_activate_delay + 48
    frame #1: tmxr_clock_coschedule_tmr + 108
    frame #2: tti_svc + 28
    frame #3: sim_process_event + 952
    frame #4: sim_instr + 1952
    frame #5: run_cmd + 2544
    frame #6: do_cmd_label + 2080
    frame #7: main + 3412

The two frames on top pin both halves of the bad state simultaneously: reaching _tmxr_activate_delay at all requires UNIT_TM_POLL to be set (the guard in frame #1), and faulting at +48 requires uptr->tmxr to be NULL.

Mechanism

1. tmxr_attach_ex is the only thing that establishes the pair — uptr->tmxr (sim_tmxr.c:4456, and per-line at 4481/4485) together with UNIT_TM_POLL (4469, 4479, 4483).
2. SAVE writes uptr->dynflags (scp.c:8658) and RESTORE reads it back (scp.c:8924). UNIT_TM_POLL therefore returns in a fresh process where uptr->tmxr is still NULL — a pointer cannot be, and is not, part of the save file.
3. sim_rest re-attaches previously attached units (scp.c:9081) but treats attach failure as non-fatal (9083): it prints and moves on, leaving the restored flag in place.
4. All three schedulers gate on the flag alone — tmxr_activate (4815), tmxr_activate_after (4848), tmxr_clock_coschedule_tmr (4899) — and call _tmxr_activate_delay (4763), which does TMXR *mp = (TMXR *)uptr->tmxr; (4765) and immediately dereferences mp->lines (4769).

tmxr_detach clears UNIT_TM_POLL alongside the pointers (4745–4759), so an ordinary detach is safe. The restore path is the hole: it re-imports the flag without re-establishing what the flag implies.

How this is hit in practice

Any workflow that suspends with SAVE and resumes with RESTORE across process restarts. If the previous incarnation's telnet pairs are still in TIME_WAIT, the listener bind fails (bind error 48) because tmxr_open_master does not set SO_REUSEADDR; the restore then proceeds with polling flags set and no mux behind them, and the first CONTINUE segfaults. The occupied-port repro above is just a deterministic stand-in for that race.

Suggested fix

Two levels, either or both:

• Defensive — in _tmxr_activate_delay, if (mp == NULL) return interval;, or have the three callers test uptr->tmxr alongside UNIT_TM_POLL. This removes the crash class wherever else a stale flag might arise.
• Root cause — make restore leave consistent state: when scp_attach_unit fails in sim_rest, clear the polling state the flag implies for that unit (UNIT_TM_POLL, and the UNIT_ATT the save file implied), so it degrades to an ordinary non-mux unit. Alternatively, mask UNIT_TM_POLL out of the restored dynflags and let a successful attach re-establish it, since it is runtime-derived state rather than saved machine state.

Raising restore-time attach failure to a visible error (or a distinct return status) would also help — currently the session looks healthy right up to the crash.

Note on a workaround that looks right and isn't

reset <dev> re-links the unit pointers as a side effect and does make the crash disappear. It also zeroes device CSR state the restored guest OS had already configured — interrupt enable included — so the restored guest silently stops seeing console input. Anyone hitting this crash should not take reset as the fix.

[markpizz]

The whole concept of SIMH's SAVE/RESTORE is generally less than perfect.

Less than perfect meaning that, in order for a simulator to behave properly, EVERY DEVICE in the simulator MUST completely describe the complete DEVICE state in the DEVICE's REGister variables. There is no way to automatically confirm that this requirement is true. I'm 100% sure that the VAX (MicroVAX3900) simulator meets this requirement since I fixed things so that this requirement was being met and I have been using SAVE/RESTORE in production almost 2 decades ago. I can't make any assertions about any other simulator, although many are likely fine.

Meanwhile, even if all DEVICE state for ALL DEVICE's in a simulator is properly described in REGisters, there is some state which describes things that can't be saved since it reflects dynamic state between the internals of the simulator and other things with the host operating system. Specific examples of this issue are connections into (or out of) the simulator between the simulator and outside activities (i.e. telnet connections to simulated serial ports). Other network connections also fall into this potentially problematic category. Since full TMXR device state can't be restored, the best we can do is merely restore whatever the TMXR ATTACH string was used, but active connections will certainly be dropped. If any TMXR connections were previously existing, the saved REGister state for that DEVICE may be inconsistent with respect to the internal state of the TMXR connection status.

However, even if any such devices aren't going to be restored to 100% functional states, the simulator should never crash.

Code in the repo at https://github.com/simh/simh doesn't crash after a RESTORE when a SET CONSOLE TELNET=xxx has been done before the SAVE. Other changes in that codebase avoided this problem (not deliberately) more than a year ago.

[ChristineTham]

Thank you — that's a genuinely clarifying answer, and it settles something I'd
been treating as a bug when it's really a documented limit of what SAVE/RESTORE
can promise. The point that TMXR state is dynamic host state, and that the best
RESTORE can do is replay the ATTACH string, is the piece I was missing.

For anyone who finds this later, here's what I changed on my side, and one thing
that may still be worth looking at independently of the crash.

What I do now. I stopped letting restore do the attaching at all:

restore -D -Q state.sav

and then attach every unit by hand. Two details turned out to matter:

• the disk is attached before the restore and the DZ after it, because
  dz_attach's lp->rcve/DTR fixup only runs when CSR_MSE is already set —
  i.e. against restored registers;
• every listener gets a fresh port per launch, since a quick relaunch otherwise
  lands on the previous incarnation's TIME_WAIT pairs.

That has been reliable, and your explanation is why: I'm no longer asking
RESTORE to reconstruct something it can't.

The part that may still be worth a look. In my case the dropped connection
wasn't what hurt — the handling of the failed attach was. In scp.c's restore
path the per-unit attach loop is gated on r == SCPE_OK and r is never reset,
so the first failed attach silently skips every remaining unit. On my
VAX-11/780 configuration the DZ precedes RP0 in device order, so a DZ port still
in TIME_WAIT meant the machine came back with no disk attached at all, while
restore reported success.

That failure is unusually hard to read from the outside. The kernel is still in
memory, so the console answers and the shell echoes its prompt; it's only when
something tries to exec that you get Killed, and login never reaches a
shell. It presents as a terminal that has gone quiet, which is nothing like
"your root filesystem is gone".

I have a small probe that demonstrates it either way — it holds the saved port so
the DZ attach must fail, then prints whether RP0 came back attached. Happy to
attach that, or to test a patch against vax780, if either would help.

On the crash itself. What I saw is consistent with your description: after a
failed console re-attach, the next cont segfaults in _tmxr_activate_delay
dereferencing uptr->tmxr. save persists UNIT_TM_POLL in the unit's
dynflags, but the uptr->tmxr pointer is only set by a successful
tmxr_attach, so the restored unit believes it is polling a multiplexer that was
never attached. Good to know that codebase no longer reaches that path.

And understood re: scope — I'm on vax780, not MicroVAX3900, so I'm not assuming
the same guarantees hold. If it's useful, I'm exercising 780 SAVE/RESTORE fairly
hard (suspend and resume on every application launch) and I'm glad to report
anything else that turns up.

Thanks again for taking the time to explain the design constraint rather than
just the symptom.

[markpizz]

A little bit of further clarification. SAVE/RESTORE on the MicroVAX3900 is not perfect either, It is still subject to the TMXR inconsistent state issues that affect all other simulators. My long time usage avoids TMXR devices for my SAVE/RESTORE activities.

You might be able to sidestep the TMXR restore complexities is to directly telnet into the SAVE/RESTORE system and let the simulated OS network stack recover the login sessions as "disconnected" which you connect to again later.

The latest code in https://github.com/simh/simh will process ALL device attach attempts and report any issues observed as they happen.

I am curious what your doing with the SAVE/RESTORE on this system.

My use of SAVE/RESTORE serves 2 purposes:

1. When the host OS is being backed up, the simulator state is saved in a save file which gets backed up along with everything else. Once the backup is done, the state is restored and execution continues.
2. When the host system is shutdown (or rebooted), the simulated state is saved and automatically restarted the next time the host system boots.

In both cases, what is running in the simulator is unaffected by the things the host is doing. The only odd effect on the simulated system is that it's clock won't be ticking during backups or when the host system isn't running. This problem is addressed by a constantly running process which sleeps for 5 minutes and then sets the running OS clock from the hardware time of year clock and then goes back to sleep.

[ChristineTham]

Happy to say what it's for — and I think your telnet suggestion points at the
real principle, even though I can't take it.

What the SAVE/RESTORE is doing

It's an iPad and Mac application. vax780 runs Research Unix 8th Edition, and
an emulated AT&T DMD 5620 terminal (a WE32100 interpreter) hangs off DZ11 line
0 — the 1985 topology, both interpreters in one process, joined by a loopback
telnet pair. What the user is looking at is the 5620's 800×1024 framebuffer.

So SAVE/RESTORE isn't a maintenance event here, it's the application
lifecycle. iOS suspends and then kills backgrounded apps whenever it likes,
so:

• background → ^E → save state.sav → paused, zero CPU (the battery
  argument; an idle login: otherwise burns a core, which is a separate
  UNIT_IDLE story)
• foreground → continue
• cold launch with a snapshot present → restore -D -Q and attach everything
  by hand

That's the difference from your two cases: yours are occasional and
host-initiated, with you present. Mine happens several times a day, unattended,
and every application launch is a restore. Cold boot to login: is
25–30 s; restore is a couple of seconds, and that gap is the whole reason the
mechanism is on the hot path at all. Snapshot is ~1.9 MB.

The one invariant I enforce hard: state.sav is deleted the moment the machine
runs again. A snapshot is only consistent with the disk while the machine stays
paused, so restoring a stale one against a disk that has moved on would corrupt
the filesystem. An unclean kill therefore cold-boots, and V8's autoboot fsck
heals it — authentic 1985 behaviour, as it happens.

On telnetting in instead

I think the general principle underneath your suggestion is where the
connection state lives. TMXR keeps per-line connection state in the
simulator's own host memory, which no REG describes and no save file can carry.
A network device keeps it in the guest's TCP stack — and that's just memory,
which SAVE covers completely. The guest can recover because the state was
inside the thing being checkpointed. That reframing is useful well beyond this
issue, and I'd not thought of it that way.

And it isn't hypothetical for me: V8 ships a telnetd
(usr/src/cmd/inet/etc/telnetd.c, Berkeley 4.26), and I have its TCP/IP stack
running over an Interlan NI1010 I modelled for SIMH, so I could do exactly what
you describe.

It still doesn't fit, because here the serial line isn't transport — it's the
product. The far end of the DZ is the other emulated machine, running AT&T's
mux windowing protocol over an 8-bit link. There's no login session to
reconnect to; there's a bitmap terminal. I'm the pathological case for
"avoid TMXR devices": the mux is the machine.

What I do instead is your trick applied to both ends at once — I checkpoint the
terminal too (its NVRAM and its screen), so the pair comes back together and
the wire between them stays pure transport with nothing worth saving.

The part that took me longest is the part your approach gets free from the
protocol: a restored serial session has nothing left to say. getty prints
its banner and login: exactly once and then blocks in getname(); a
logged-in shell prints nothing unasked. A cold boot gets a prompt for free, a
restore never does, so the terminal has to send its own CR on resume — that
nudge is the only thing that makes the resumed far end speak. Miss it and you
have a photograph of a terminal.

Your clock daemon — and a V8 wrinkle you'd enjoy

I went and read why your resync works, and it's better than I assumed:
todr_rd() in VAX/vax_stddev.c isn't returning a counter at all in the
default mode. It computes now − toy_gmtbase from the host clock on every
read, so the TODR genuinely never stops — not across a save/restore gap, not
across host downtime. Your daemon isn't correcting drift, it's harvesting a
register that was never wrong.

On V8 that recovers the time of year and not the year. clkinit() in
sys/sys/machdep.c takes only (todr − TODRZERO)/100 — seconds into the year
— and then gets the year by advancing from the root superblock's time in
whole-year steps until the two are within six months:

for (time = (todr-TODRZERO)/100; time < base-SECYR/2; time += SECYR) {
        if (LEAPYEAR(year))
                time += SECDAY;
        year++;
}

TODRZERO is 1<<28 and YRREF is 1970 (sys/h/clock.h). So the year lives
in the filesystem, not in the hardware, and no amount of TODR correctness
reaches it. If the superblock time is below 5*SECYR you get

WARNING: preposterous time in file system -- CHECK AND RESET THE DATE!

and the clock is jammed to the hardcoded 6*SECYR + 186*SECDAY + SECDAY/2 —
which is why so many V8 images sit in mid-1976 and blame the simulator. I did,
for a while.

date(1) can't set it out of that either: Berkeley's 1980 date.c does a bare
year += 1900 on the two digits it reads, so 26 means 1926 — before YRREF,
so the year loop above contributes nothing. I had to add the usual 69/70 pivot.

None of which is cosmetic for me: I'm building V8 from source inside the
emulator, and make(1) compares mtimes. A wrong clock doesn't look like a
wrong clock — it silently changes what gets rebuilt.

So the V8 equivalent of your daemon is your daemon plus stime(2) in GMT and a
sync, so the superblock carries the year forward to the next boot. The TODR
half works exactly as you described it.

On the attach loop

Confirmed on my side: open-simh's scp.c still has

for (j=0, r = SCPE_OK; j<attcnt; j++) {
    if ((r == SCPE_OK) && (!dont_detach_attach)) {

with r never reset, so the first failure still skips every remaining unit.
Good to know the code at simh/simh processes them all and reports as it goes —
that behaviour alone would have turned my "machine came back with no disk"
into a one-line diagnosis instead of a day of it. I have a note to go through
what else there is worth carrying across.

Thanks for the detail; this was more useful than a fix would have been.