Concurrent background pollers with no shared circuit breaker cause console reboot crash-loop

[Bagomojo]

Environment

• CloudKey Gen2, ~2.9GB RAM)
• • Controller version 10.6.101-35991
• • • NetworkOptimizer.Web running as a systemd service

Symptom

Console entered a reboot crash-loop (multiple reboots within minutes of each other, unclean shutdowns) starting immediately after NetworkOptimizer was installed and pointed at the console's API.

Root cause

UniFiApiClient.cs has a bounded, sane per-call retry policy (3 attempts on connection failure with 2/4/8s exponential backoff, plus 1 retry on a 15s timeout) - see the WrapAsync(transientRetry, timeoutRetry) policy around line 154-161. That's fine in isolation.

The problem is there's no shared circuit breaker / "controller is currently unreachable" state consulted across callers. When the console blips even briefly, every independent background service pays its own full retry cost against the same target at the same time, with no coordination between:

• The dashboard's own data refresh (network configs, devices, wireless clients)
• • ApAgentRegistry's supervise loop (2 min interval)
• • • SqmService's WAN/status check (2 min interval), including an SSH connection attempt to the gateway
• • • • Traffic-flow / threat-log polling (~30s interval)

Log evidence

The instant the console becomes unreachable, multiple subsystems fail and begin retrying independently within the same second:

Sep 04 23:38:47 NetworkOptimizer.Web[12110]: [23:38:47 WRN] Failed to fetch network configs
Sep 04 23:38:47 NetworkOptimizer.Web[12110]: System.Net.Http.HttpRequestException: Connection refused (192.168.2.2:443)
Sep 04 23:38:47 NetworkOptimizer.Web[12110]: [23:38:47 ERR] Failed to fetch access points
Sep 04 23:38:47 NetworkOptimizer.Web[12110]: System.Net.Http.HttpRequestException: Connection refused (192.168.2.2:443)
Sep 04 23:38:47 NetworkOptimizer.Web[12110]: [23:38:47 ERR] Failed to fetch wireless clients
Sep 04 23:38:47 NetworkOptimizer.Web[12110]: System.Net.Http.HttpRequestException: Connection refused (192.168.2.2:443)
Sep 04 23:38:47 NetworkOptimizer.Web[12110]: [23:38:47 WRN] Retry 1 after 2s due to Connection refused (192.168.2.2:443)
Sep 04 23:38:49 NetworkOptimizer.Web[12110]: [23:38:49 WRN] Retry 2 after 4s due to Connection refused (192.168.2.2:443)
Sep 04 23:38:51 NetworkOptimizer.Web[12110]: [23:38:51 WRN] Retry 3 after 8s due to Connection refused (192.168.2.2:443)
Sep 04 23:38:54 NetworkOptimizer.Web[12110]: [23:38:54 WRN] AP Agent supervision pass failed
Sep 04 23:39:11 NetworkOptimizer.Web[12110]: [23:39:11 ERR] Error fetching WAN interfaces from controller
Sep 04 23:39:17 NetworkOptimizer.Web[12110]: [23:39:17 WRN] Unhandled exception rendering component: UniFi Console returned non-JSON content (status 200, content-type text/html) - it may be rebooting or mid-firmware-upgrade.
Sep 04 23:39:17 NetworkOptimizer.Web[12110]: [23:39:17 ERR] Unhandled exception in circuit 'MdOfk7Ynea8IIp0RdFDmAogh-c8cCyQ_Y4P3C17C5K8'.
Sep 04 23:39:45 NetworkOptimizer.Web[12110]: [23:39:45 ERR] SSH command timed out for 192.168.2.2: Connection failed to establish within 5000 milliseconds.
Sep 04 23:39:53 NetworkOptimizer.Web[12110]: [23:39:53 WRN] Retry 1 after 2s due to An error occurred while sending the request.
Sep 04 23:39:53 NetworkOptimizer.Web[12110]: [23:39:53 WRN] Retry 1 after 2s due to An error occurred while sending the request.
Sep 04 23:39:55 NetworkOptimizer.Web[12110]: [23:39:55 WRN] Retry 2 after 4s due to Connection refused (192.168.2.2:443)
Sep 04 23:39:55 NetworkOptimizer.Web[12110]: [23:39:55 WRN] Retry 2 after 4s due to Connection refused (192.168.2.2:443)

This continues in overlapping waves (independent retry cycles from separate callers) for several minutes past 23:44:50 before the console stabilizes.

Separate bug, same area

"Traffic flows" and "Threat log events" requests return 405 Method Not Allowed on every poll, indefinitely, regardless of controller health - wrong endpoint/method for this controller version:

Sep 04 23:41:29 NetworkOptimizer.Web[12110]: [23:41:29 WRN] Traffic flows request failed: MethodNotAllowed - {"error":{"code":405,"message":"Method Not Allowed"}}
Sep 04 23:41:29 NetworkOptimizer.Web[12110]: [23:41:29 WRN] Threat log events request failed: MethodNotAllowed - {"error":{"code":405,"message":"Method Not Allowed"}}
Sep 04 23:41:59 NetworkOptimizer.Web[12110]: [23:41:59 WRN] Traffic flows request failed: MethodNotAllowed - {"error":{"code":405,"message":"Method Not Allowed"}}
Sep 04 23:42:29 NetworkOptimizer.Web[12110]: [23:42:29 WRN] Traffic flows request failed: MethodNotAllowed - {"error":{"code":405,"message":"Method Not Allowed"}}

Not tied to the crash-loop, but constant wasted load and error-log noise every 30 seconds indefinitely.

Suggested fix

Add one shared circuit-breaker state (e.g. a static "controller unreachable since " flag with its own cooldown) that every subsystem - dashboard refresh, ApAgentRegistry, SqmService, traffic-flow/threat-log poller - checks before firing a request, so the whole app backs off together instead of each piece retrying independently against a console that's still booting.

[tvancott42]

Did you attempt to install NO on the CloudKey G2 itself? I honestly wouldn't recommend this ever unless you're just trying to experiment. It doesn't have nearly enough RAM to host NO. CPU power, maybe (IIRC it's 8 cores ARM Cortex, a couple gen old).

The full chain of events is a little unclear here, so paint it out for me so I can better understand what kicked off the crash loop. Let's start with where NO is deployed, and if any events, restarts, upgrades, etc. were happening on the UCK at the same time. I'm unclear on this from the description.

Also on "wrong endpoint/method for this controller version" - these endpoints don't exist at all on UniFi Network on UCK, I believe. I can cache this state for the lifetime of the console connection/config; it is definitely an edge case that should be accounted for, but "wasted load" is a bit of an overclaim here, it's almost infinitesimal even in aggregate for the Console to kick back 405 errors from its middleware.

Your findings are correct on a crash-looping UniFi Console and coordinated retries, but let's narrow down the actual root cause here first. We've vetted the console-upgrading / console-rebooting lifecycle ten thousand times over in this product, so what we've got here is an edge case.

The real impact of uncoordinated retries really is negligible as they're just either refused at the socket level (physically zero Console impact) or hitting a static HTML page on the Console until it's actually up, then everything flows per normal. The root cause here is what we really need to pinpoint.

The only other thing I can see in your logs is attempted SSH to the CloudKey Console and not your UniFi Gateway, which is configuration you'll want to correct once you get this sorted out.