Skip to content

Cluster runtime retains targets while waiting for a lease #1025

Description

@pood1e

Bug

In a clustered subscribe deployment, GET /api/v1/targets can retain the same target on two members after a lease changes owner. The lease has one holder, but a member waiting to acquire it inserts the target into its runtime map before Lock returns. If another member keeps the lease, that pending entry can remain indefinitely. A lease-loss retry can create another pending entry through the same path.

Impact

Runtime target counts and membership checks report duplicates even though the lease limits active subscription ownership. Pending goroutines and target objects may also remain until the lease becomes available or the process restarts.

Expected behavior

A lock attempt should have a finite deadline. Failed or canceled attempts and completed lock sessions should remove their runtime target generation. Periodic assignment reconciliation can retry when ownership changes.

This is separate from the loader snapshot deletion case in #1023.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions