Describe the bug
In clustered collect mode with an HTTP target loader, a target can remain configured on a member after it disappears from the discovery response. httpLoader.updateTargets updates lastTargets before the cluster has confirmed that the member processed the deletion. If a member's DELETE request fails once, subsequent successful HTTP polls have no diff, so the loader never sends the deletion again. The cluster DELETE path also accepts a non-2xx HTTP response without returning an error.
Reproduction
- Run a clustered collector with an HTTP loader that initially returns one target.
- Change the HTTP response to an empty target map.
- Make the member's first
DELETE /api/v1/config/targets/{name} return 503 (or interrupt the leader while handling the delete).
- Restore the member API and let discovery poll several more times.
The target can remain in the member's config and continue collecting even though the HTTP source no longer includes it. A leader restart can also lose the loader's in-memory deletion diff while a member retains the target.
Expected behavior
Successful discovery snapshots should converge member target configs to the source across transient member failures and leader changes. HTTP 5xx on deletion should be reported and retried; 404 can be treated as already deleted.
Relevant code
pkg/loaders/http_loader/http_loader.go: updateTargets advances lastTargets and emits only non-empty diffs.
pkg/app/loaders.go: the cluster delete error is logged but does not feed back into the loader.
pkg/app/clustering.go: deleteTarget does not validate the HTTP response status.
No private infrastructure or target data is needed to reproduce this.
Describe the bug
In clustered
collectmode with an HTTP target loader, a target can remain configured on a member after it disappears from the discovery response.httpLoader.updateTargetsupdateslastTargetsbefore the cluster has confirmed that the member processed the deletion. If a member's DELETE request fails once, subsequent successful HTTP polls have no diff, so the loader never sends the deletion again. The cluster DELETE path also accepts a non-2xx HTTP response without returning an error.Reproduction
DELETE /api/v1/config/targets/{name}return 503 (or interrupt the leader while handling the delete).The target can remain in the member's config and continue collecting even though the HTTP source no longer includes it. A leader restart can also lose the loader's in-memory deletion diff while a member retains the target.
Expected behavior
Successful discovery snapshots should converge member target configs to the source across transient member failures and leader changes. HTTP 5xx on deletion should be reported and retried; 404 can be treated as already deleted.
Relevant code
pkg/loaders/http_loader/http_loader.go:updateTargetsadvanceslastTargetsand emits only non-empty diffs.pkg/app/loaders.go: the cluster delete error is logged but does not feed back into the loader.pkg/app/clustering.go:deleteTargetdoes not validate the HTTP response status.No private infrastructure or target data is needed to reproduce this.