Skip to content

Vulnerability metrics: don't re-query a stalled database on every scrape - #1086

Merged
javuto merged 1 commit into
developfrom
vulnerability-monitoring-metrics-backoff
Oct 11, 2026
Merged

javuto merged 1 commit into
developfrom
vulnerability-monitoring-metrics-backoff

Conversation

@javuto

@javuto javuto commented Oct 11, 2026

Copy link
Copy Markdown
Collaborator

Vulnerability metrics: don't re-query a stalled database on every scrape

Follow-up to #1082, which left this as a known limitation.

Problem

The vulnerability gauges read the shared database, and a reading is reused for one minute. A reading that failed was not remembered. While the database was stalled, every Prometheus scrape started a new query and waited up to the 10-second deadline. The scrapes also queued behind each other on the collector lock.

What changes

  • Failure backoff (pkg/vulns/metrics.go): the collector records when a read failed. For one minute (metricsTTL) after that, it skips the database and exports nothing.
    • A failed read still exports nothing, never zeros, which could read as "no findings".
    • After the minute, the next scrape tries again.
    • A successful read clears the failure.
  • No other behaviour changes. The gauges, their labels and the success-path caching are unchanged.

Security and operational impact

  • Flags and schema: none. No new flags, no schema change, no API change. The change only affects osctrl-tls when --vuln-enabled is on.
  • Stalled database: scrapes return at once instead of each waiting up to 10 s.
  • Recovery: after the database recovers, the gauges can take up to one minute to reappear. This is the same cadence as the normal reading cache.

Validation

  • go test ./pkg/vulns -count=1: passes
  • gofmt -l pkg/vulns (empty) and go vet ./pkg/vulns: clean
  • New test: TestMetricsFailedReadIsNotRetriedWithinTheTTL.
    • A dropped table makes the first read fail.
    • The table is then restored.
    • A scrape inside the TTL still exports nothing, and one after the TTL exports the data.
    • The test failed without the fix.
  • No frontend or OpenAPI changes, so those checks don't apply.

@javuto javuto added the 📊 metrics Metrics related issues label Oct 11, 2026
@javuto
javuto merged commit 9a3d441 into develop Oct 11, 2026
7 checks passed
@javuto
javuto deleted the vulnerability-monitoring-metrics-backoff branch October 11, 2026 18:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

📊 metrics Metrics related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant