Problem
The Kubernetes storage garbage collector calls listN(..., 500) for expiring resources but does not follow the Kubernetes metadata.continue token. Once more than 500 objects exist, objects outside the first page can remain indefinitely even when their stored expiry is in the past.
Current source (master at ab64ed778070e983cbb10cfc07ea4bb397d14312):
We observed this with Dex v2.42.0 and Kubernetes storage: more than 13,000 AuthRequest resources accumulated despite expiry.authRequests: 24h; more than 10,000 had a stored expiry older than 24 hours. The growing object set adds API-server/etcd watch and cache pressure.
Expected behavior
Kubernetes-backed GC should eventually inspect all eligible objects while bounding work per cycle. Possible approaches:
- follow
metadata.continue until the configured per-cycle deletion budget is exhausted; or
- persist/rotate a continuation cursor between GC cycles so a bounded scan cannot remain pinned to the first page.
Deletion should continue to use the resource's stored expiry, and a disappearing object during GC should be treated as a benign race.
Local mitigation
We are adding a narrowly scoped, GitOps-managed maintenance job that initially reports only, paginates the AuthRequest list, and can later delete a small rate-limited batch only after re-reading the resource and confirming its stored expiry. We would prefer to retire that workaround once upstream GC is pagination-safe.
Problem
The Kubernetes storage garbage collector calls
listN(..., 500)for expiring resources but does not follow the Kubernetesmetadata.continuetoken. Once more than 500 objects exist, objects outside the first page can remain indefinitely even when their stored expiry is in the past.Current source (master at
ab64ed778070e983cbb10cfc07ea4bb397d14312):GarbageCollectlimits each resource list to 500: https://github.com/dexidp/dex/blob/ab64ed778070e983cbb10cfc07ea4bb397d14312/storage/kubernetes/storage.go#L1241-L1249limitand does not consume continuation tokens: https://github.com/dexidp/dex/blob/ab64ed778070e983cbb10cfc07ea4bb397d14312/storage/kubernetes/client.goWe observed this with Dex v2.42.0 and Kubernetes storage: more than 13,000
AuthRequestresources accumulated despiteexpiry.authRequests: 24h; more than 10,000 had a storedexpiryolder than 24 hours. The growing object set adds API-server/etcd watch and cache pressure.Expected behavior
Kubernetes-backed GC should eventually inspect all eligible objects while bounding work per cycle. Possible approaches:
metadata.continueuntil the configured per-cycle deletion budget is exhausted; orDeletion should continue to use the resource's stored expiry, and a disappearing object during GC should be treated as a benign race.
Local mitigation
We are adding a narrowly scoped, GitOps-managed maintenance job that initially reports only, paginates the
AuthRequestlist, and can later delete a small rate-limited batch only after re-reading the resource and confirming its stored expiry. We would prefer to retire that workaround once upstream GC is pagination-safe.