IWDT panic in ml_wg_mgr during the peer-table NVS flush (v1.3 fleet firmware coredump)

[iliabaranov]

Evidence

Remote 7a60 retained a coredump from the v1.3 release firmware (fw_sha b26e155bc), decoded with the archived ELF:

• exccause 0x46 InterruptWDTTimeoutCPU1, crashed task ml_wg_mgr, uptime 9598 s.
• Stack: ml_wg_mgr_task → ml_peer_nvs_flush_if_due → flush_table → nvs_set_blob("tbl", 11780 B) → nvs::Storage::writeMultiPageBlob → esp_flash_write(3968 B) → spi_flash_disable_interrupts_caches_and_other_cpu (cache_utils.c:176, spinning) — the other core was parked in spi_flash_op_block_func (cache_utils.c:109).
• The reboot that follows is a remote drop / STOP window for the machine it is bonded to.

Why it matters

ml_peer_nvs.c flushes the whole peer table (~11.8 KB blob, multi-page) from inside the WireGuard manager task (the safety data-plane task) at most every 5 s when dirty. A multi-page NVS write stalls both cores for the flash operation; once it exceeded the 300 ms interrupt-watchdog budget (CONFIG_ESP_INT_WDT_TIMEOUT_MS=300). Soak run 4 (8 h, three units) measured flush durations of 103–152 ms on every unit (nvs_pf telemetry from #154) — routinely above the 80 ms comparator publish budget and within a factor of ~2 of the IWDT limit, so the panic is a tail event of a normal operation, not an anomaly.

Not yet root-caused: what made this particular write take > 300 ms (candidates: the IPC handshake to the other core delayed by a long critical section, or an erase folded into the write when a page filled). No reproduction yet.

Proposed follow-ups

1. Move the flush off ml_wg_mgr onto a low-priority worker (it is durability, not data plane).
2. Chunk or rate-limit the blob write (per-entry records or smaller pages) so no single flash op approaches the IWDT budget; consider CONFIG_SPI_FLASH_YIELD_DURING_ERASE interaction.
3. Keep exposing nvs_pf (last/max/count/start) — it is the gauge for (1)/(2).

Coredump + decode archived under ops/coredumps/2026-09-19-7a60-ml_wg_mgr-v1.3-b26e155bc.* (local ops archive; the raw dump is not attached here because it embeds RAM contents).

[iliabaranov]

Status 2026-09-21: v1.3.1 exposes the gauge for this (nvs_pf = last/max flush duration in /state.json, plus nvs_dcs for the dcs-side writes). Across the 8 h and 1 h soaks: peer-cache flushes take 103–152 ms on all three units, every 5 s when the table is dirty; dcs-side writes 3–7 ms. No IWDT panic recurred on the bench, but the flush is within ~2× of the 300 ms budget on every unit, so the v1.3 coredump was a tail event of normal operation, not a one-off.

Plan (unchanged, now the next firmware item after the release soak): move the flush off ml_wg_mgr onto a low-priority worker, and chunk the 11.8 KB blob so no single flash op approaches the IWDT budget. Candidate for v1.3.2.