Prometheus Write backpressure retains per-target responses and can OOM subscribe mode

[pood1e]

Bug description

In subscribe mode, slow output transformation can retain up to target count × target buffer size complete gNMI SubscribeResponse protobufs. With many targets or large initial snapshots, memory grows until the process is OOM-killed.

The reproduced path is:

gRPC Recv
  -> Target.handleStreamSubscriptionRcv
  -> per-target subscribeResponses channel
  -> App.StartTargetsManager
  -> App.export
  -> prometheus_write.Write
  -> unbuffered msgChan
  -> limited output workers

Each target's receive channel defaults to 100 responses. App.export waits for the selected outputs, while prometheus_write.Write waits for an available worker through an unbuffered channel. When event processing is slower than incoming initial snapshots, every target independently fills its receive buffer with complete protobuf payloads.

After the output timeout, prometheus_write.Write returns and drops the ingress message. That drop is not included in the Remote Write failure metric because no HTTP request was attempted.

This is not an unreachable-object leak. It is multiplicative, reachable retention caused by the absence of a process-wide bounded input queue and explicit overload accounting.

Type

• Performance defect
• Reliability defect
• Design defect

Reproduction

Tested in subscribe mode with main at c820a545e8c32325e15c5a8cd595965f9e57ae32:

1. Start 220 active target entries backed by 20 simulators.
2. Configure prometheus_write with 2 workers, 2 writers and a 10,000 time-series buffer.
3. Point Remote Write at a local endpoint that immediately returns HTTP 204, excluding network and remote storage latency.
4. Let all 220 streams reach READY and emit their initial snapshots.

Observed growth:

Sample	Heap alloc	Heap sys	RSS	Goroutines	FDs
Initial	2.34 GB	2.85 GB	2.71 GB	2019	228
+44 seconds	3.17 GB	5.39 GB	5.23 GB	2018	228
+2 minutes	5.01 GB	5.98 GB	5.85 GB	2018	228

A heap profile reported 2.61 GB in use, with 2.40 GB (92.02%) retained below protobuf consumeBytes on the gRPC Recv -> Target.handleStreamSubscriptionRcv path. The goroutine profile showed target listeners blocked in promWriteOutput.Write while two workers transformed messages.

The Remote Write endpoint returned 204 and its send-failure metric stayed at zero. The retained objects are therefore upstream of HTTP Remote Write.

The same failure at larger scale becomes a repeated OOM/restart/failover loop. Adding memory only delays the failure because retained data grows with the backlog.

Expected behavior

• prometheus_write has a process-wide bounded input queue shared by all targets.
• Queue capacity, current depth and dropped ingress messages are observable.
• Output overload cannot fill 100 response slots independently for every target.
• Dropped ingress messages are distinct from HTTP Remote Write failures.
• Memory remains bounded when input rate exceeds transformation throughput.

Actual behavior

• prometheus_write.msgChan is unbuffered.
• Target listeners wait up to the output HTTP timeout for a worker.
• Each target has an independent 100-response receive buffer.
• Ingress timeouts return silently unless debug logging is enabled.
• Remote Write failure metrics remain zero for messages dropped before HTTP delivery.

Impact

• Affected component: subscribe mode with CPU-heavy processors and prometheus_write.
• Affected versions: reproduced on current main; the underlying behavior has existed across multiple releases.
• User impact: collector OOM, repeated HA failover, missing telemetry and misleading delivery metrics.
• Severity: high. A temporary output throughput reduction can take every collector replica out of service.

Introduction history

There is no single regression commit; the failure is the interaction of three changes:

• Commit 664f1a7e (2022-04-11) introduced the default per-target receive buffer of 100.
• add a configurable number of workers to prometheus outputs #235 / commit c056a92f (2023-10-03) introduced the shared unbuffered Prometheus input channel, configurable workers and the enqueue timeout. It fixed I think Prometheus output have a memory leak that crashes the server it runs on.  #234 for the tested rate but left ingress capacity and drops unobservable.
• Simplify interfaces #746 / commit 41f58027 (2025-09-22) introduced the current App.export implementation, which waits for output writes per target and therefore exposes the multiplicative target buffers when workers are saturated.

While investigating this path, a separate collector-mode problem was found: #774 introduced goroutine-per-message output dispatch and later raised its pipeline capacity to 1,000,000. That path is not used by this reproduction and should be handled separately.

Related: #234.

[pood1e]

Proposed solution

Fix the reproduced prometheus_write ingress boundary:

1. Add a configurable, process-wide input-buffer-size for protobuf messages before event conversion.
2. Make Write a bounded admission operation. When the input queue is full, return immediately instead of retaining one blocked output call per target.
3. Export stable metrics for input queue capacity, current depth and dropped ingress messages, with a bounded reason label.
4. Keep the existing time-series buffer-size, num-workers and num-writers settings: they control different stages and remain independently tunable.
5. Preserve runtime output updates by safely replacing and draining the input channel when input-buffer-size changes.
6. Add concurrent update and saturation tests proving that retained input messages never exceed the configured capacity, calls remain non-blocking under overload, drops are counted and shutdown remains prompt.

This removes the multiplicative retained backlog from the reproduced path. It does not hide sustained overload: operators can increase transformation workers/CPU, reduce input rate, or alert on the new drop counter.

Cost

• prometheus_write configuration, lifecycle and metrics: moderate change.
• Documentation and regression tests: small change.

Benefit

• Establishes a deterministic input-queue bound shared by all targets.
• Prevents processor saturation from turning into process OOM.
• Makes pre-HTTP telemetry loss observable instead of reporting zero Remote Write failures.
• Reuses the bounded-buffer pattern already present in other gNMIc outputs.

[pood1e]

Implemented in #948 (commit 0012c89c).

Validation completed locally:

• ./tests/run_tests.sh
• go test -race -count=20 ./pkg/outputs/prometheus_output/prometheus_write_output
• go vet ./... in the root, pkg/api, and pkg/cache modules
• git diff --check

All commands passed.

[pood1e]

Implementation review update

The original drop-on-full prototype was superseded by commit c13ec4c4 in
#948. Dropping at this boundary avoids blocking but silently loses telemetry;
resizing and draining a live channel can also lose or reorder messages.

The revised implementation uses a bounded queue with cancellable backpressure:

• admission waits for capacity instead of dropping;
• the default per-target subscription response buffer is reduced from 100 to
  1, so backpressure reaches the gNMI stream before a large per-target backlog
  forms;
• target removal, parent cancellation, and output shutdown release blocked calls;
• downstream event and time-series handoffs are cancellable;
• input-buffer-size defaults to 1 and cannot be resized at runtime;
• queue depth/capacity and backpressure count/duration are exported.

Validation completed on the revised head:

• go test -race ./...
• git diff --check

[pood1e]

Implementation scope update

Maintainer review identified that the target response buffer is shared upstream of output dispatch and also sizes the target error channel. Changing its default in a prometheus_write-specific PR would therefore affect every output and is removed from #948.

Revised changes

• keep the existing target buffer-size default of 100
• retain the bounded prometheus_write input queue and cancellable backpressure
• document the independent output-local, target, and time-series buffers
• allow deployments to opt into a smaller target buffer explicitly
• keep this issue open because the default per-target retention multiplier remains

Cost and benefit

• Cost: small output-local configuration, lifecycle, metrics, tests, and documentation changes
• Benefit: removes silent timeout drops and makes output-local pressure bounded and observable without changing unrelated outputs
• Remaining risk: sustained output pressure can still fill each target's configured response buffer

Verification and rollback

• Verified head f277d74cf98d7a6aa011c7d9ac53ee9b717141e9 with go test -race ./... and git diff --check
• Rollback is limited to the output-local queue/cancellation changes; target buffering remains unchanged

A shared admission mechanism or global default change requires a separate design review.