You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Support Manual Auto Exit for Maintenance Mode #3033
When a cluster enters maintenance mode, partition movement is paused and only state maintenance is performed. Maintenance mode can be triggered manually (via user config) or automatically by the Helix controller if the number of offline nodes exceeds a threshold.
Maintenance mode is frequently used to batch multiple config changes, ensuring the Helix controller only recalculates assignments after all updates are complete. The workflow is:
Enter maintenance mode
Apply config changes
Exit maintenance mode
However, if an operation script fails mid-process, the cluster may remain stuck in maintenance mode, impacting availability.
Proposed Solution
Introduce a timeout parameter for manual maintenance mode entry.
When entering maintenance mode manually, users can specify a timeout.
If the cluster is still in maintenance mode after the timeout, Helix will automatically exit maintenance mode.
If no timeout is specified, auto-exit does not occur.
For controller-triggered maintenance mode (due to offline nodes), the existing behavior remains: the cluster exits maintenance mode only when the offline node count drops below the threshold.
Activity
LZD-PratyushBhatt commentedon May 12, 2025
This is dependent on #3041