Skip to content

Clustered HTTP loader does not retry target deletion after member failure #1023

Description

@pood1e

Describe the bug

In clustered collect mode with an HTTP target loader, a target can remain configured on a member after it disappears from the discovery response. httpLoader.updateTargets updates lastTargets before the cluster has confirmed that the member processed the deletion. If a member's DELETE request fails once, subsequent successful HTTP polls have no diff, so the loader never sends the deletion again. The cluster DELETE path also accepts a non-2xx HTTP response without returning an error.

Reproduction

  1. Run a clustered collector with an HTTP loader that initially returns one target.
  2. Change the HTTP response to an empty target map.
  3. Make the member's first DELETE /api/v1/config/targets/{name} return 503 (or interrupt the leader while handling the delete).
  4. Restore the member API and let discovery poll several more times.

The target can remain in the member's config and continue collecting even though the HTTP source no longer includes it. A leader restart can also lose the loader's in-memory deletion diff while a member retains the target.

Expected behavior

Successful discovery snapshots should converge member target configs to the source across transient member failures and leader changes. HTTP 5xx on deletion should be reported and retried; 404 can be treated as already deleted.

Relevant code

  • pkg/loaders/http_loader/http_loader.go: updateTargets advances lastTargets and emits only non-empty diffs.
  • pkg/app/loaders.go: the cluster delete error is logged but does not feed back into the loader.
  • pkg/app/clustering.go: deleteTarget does not validate the HTTP response status.

No private infrastructure or target data is needed to reproduce this.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions