cluster-etcd-operator deadlocks on currentRevision: Invalid value: cannot be unset once set when etcd cannot reach its peers
Version
- OKD
4.22.0-okd-scos.10 (SCOS / CentOS Stream CoreOS)
- Kubernetes
v1.35.5 (nodes) / v1.35.6 (API server)
- Platform: AWS, UPI,
x86_64
credentialsMode: Manual (ccoctl + OIDC)
Also reproduced on 4.22.0-okd-scos.8 and 4.22.0-okd-scos.9.
Summary
When etcd static pods cannot start for an unrelated reason (in our case a
network path was blocked), cluster-etcd-operator enters a hard loop it cannot
recover from. TargetConfigController fails every sync with a CEL
validation error and can never write status, so the rollout is pinned at
revision 0 permanently.
The operator does not just fail to make progress — it becomes unable to record
any nodeStatus update, which masks the real underlying fault and makes the
failure very hard to diagnose.
The error
Repeating roughly 2.5 times per second (477 occurrences in a 200-line log tail):
E0916 08:17:10.978798 1 base_controller.go:277] "Unhandled Error"
err="\"TargetConfigController\" controller failed to sync \"\",
err: Etcd.operator.openshift.io \"cluster\" is invalid:
[status.nodeStatuses[0].currentRevision: Invalid value: cannot be unset once set,
status.nodeStatuses[1].currentRevision: Invalid value: cannot be unset once set,
status.nodeStatuses[2].currentRevision: Invalid value: cannot be unset once set]"
Cluster state while deadlocked
oc get etcd cluster -o yaml:
status:
latestAvailableRevision: 1
nodeStatuses:
- nodeName: ip-10-0-10-10.ec2.internal
currentRevision: 0
targetRevision: 1
- nodeName: ip-10-0-11-11.ec2.internal
currentRevision: 0
targetRevision: 0
- nodeName: ip-10-0-12-12.ec2.internal
currentRevision: 0
targetRevision: 0All three control plane nodes were Ready. 10 other cluster operators reached
Available=True. Only etcd was stuck.
Analysis
The CRD carries three CEL validations on nodeStatuses items:
- rule: has(self.currentRevision) || !has(oldSelf.currentRevision)
message: cannot be unset once set
fieldPath: .currentRevision
- rule: oldSelf.hasValue() || !has(self.currentRevision)
message: currentRevision can not be set on creation of a nodeStatus
optionalOldSelf: true
- rule: oldSelf.hasValue() || !has(self.targetRevision)
message: targetRevision can not be set on creation of a nodeStatus
optionalOldSelf: true
The first rule is the one that fires.
CurrentRevision int32 carries json:"currentRevision,omitempty" in
operator/v1/types.go. An int32 with omitempty serialises to nothing when
its value is 0. So when the stored value is 0:
has(oldSelf.currentRevision) → false in the stored object... except the
field is materialised as 0 in the persisted resourcehas(self.currentRevision) → false for the incoming update
The transition rule therefore evaluates false and the write is rejected. The
operator cannot advance currentRevision from 0 to 1, because writing the
status at all is refused while the value sits at 0.
omitempty was added to this field in the same change that introduced these
CEL rules (4.17 → 4.18). Branches release-4.18 through release-4.22 all
carry both; release-4.17 has neither and uses json:"currentRevision"
without omitempty.
Related prior fixes for the same class of problem:
Note that library-go's node_controller.go applies nodeStatus via SSA with
applyoperatorv1.NodeStatus().WithNodeName(...) and no currentRevision
field, so whichever field manager wins ownership can drop the field entirely.
That looks like a plausible mechanism for reaching the 0/unset state, though
we did not confirm it directly.
Reproduction
The trigger in our case was that etcd static pods could not start, because the
masters could not reach the bootstrap etcd member on 2379/2380 (our own
security group misconfiguration — not an OKD bug). But the operator's response
to that condition is the issue being reported: rather than surfacing the
underlying fault, it wedges itself on a validation error it can never satisfy.
- Install OKD 4.22 UPI on AWS
- Cause etcd static pods to fail to start (blocking master → bootstrap etcd
on 2379/2380 is one way)
cluster-etcd-operator begins emitting the CEL error continuously- Observe that the error persists and the rollout stays at revision 0
Workaround
Removing the CEL validations lets the operator write a status where
currentRevision is genuinely unset:
oc patch crd etcds.operator.openshift.io --type=json \
-p='[{"op":"replace","path":"/spec/versions/0/schema/openAPIV3Schema/properties/status/properties/nodeStatuses/items/x-kubernetes-validations","value":[]}]'CVO reverts the CRD within about 45 seconds, but that window is enough. After
the patch, rejections dropped from ~150/min to 0 and stayed at 0 even once
CVO restored the rules — because the stored state was then unset rather than
0, which satisfies the transition rule.
Outcome
After fixing the underlying network issue and applying the CRD patch, the
cluster completed successfully. All three nodes are now at currentRevision: 8
with zero rejections.
We cannot say with certainty whether the CEL deadlock would have cleared on its
own once the network was fixed, or whether the patch was required. What is
clear is that while etcd pods could not start, the operator was stuck in a loop
it could not exit, and the validation error completely obscured the real cause.
Suggested fix
Either:
- Drop
omitempty from CurrentRevision int32 so 0 serialises explicitly
and has() behaves as the CEL rules assume, or
- Adjust the transition rule to tolerate the
0/unset ambiguity, e.g.
!has(oldSelf.currentRevision) || oldSelf.currentRevision == 0 || has(self.currentRevision)
Option 1 matches the pre-4.18 behaviour.
Full operator logs, the Etcd CR, and the CRD as captured during the
deadlock are available on request.
cluster-etcd-operator deadlocks on
currentRevision: Invalid value: cannot be unset once setwhen etcd cannot reach its peersVersion
4.22.0-okd-scos.10(SCOS / CentOS Stream CoreOS)v1.35.5(nodes) /v1.35.6(API server)x86_64credentialsMode: Manual(ccoctl + OIDC)Also reproduced on
4.22.0-okd-scos.8and4.22.0-okd-scos.9.Summary
When etcd static pods cannot start for an unrelated reason (in our case a
network path was blocked),
cluster-etcd-operatorenters a hard loop it cannotrecover from.
TargetConfigControllerfails every sync with a CELvalidation error and can never write status, so the rollout is pinned at
revision 0 permanently.
The operator does not just fail to make progress — it becomes unable to record
any nodeStatus update, which masks the real underlying fault and makes the
failure very hard to diagnose.
The error
Repeating roughly 2.5 times per second (477 occurrences in a 200-line log tail):
Cluster state while deadlocked
oc get etcd cluster -o yaml:All three control plane nodes were
Ready. 10 other cluster operators reachedAvailable=True. Only etcd was stuck.Analysis
The CRD carries three CEL validations on
nodeStatusesitems:The first rule is the one that fires.
CurrentRevision int32carriesjson:"currentRevision,omitempty"inoperator/v1/types.go. Anint32withomitemptyserialises to nothing whenits value is
0. So when the stored value is0:has(oldSelf.currentRevision)→ false in the stored object... except thefield is materialised as
0in the persisted resourcehas(self.currentRevision)→ false for the incoming updateThe transition rule therefore evaluates false and the write is rejected. The
operator cannot advance
currentRevisionfrom 0 to 1, because writing thestatus at all is refused while the value sits at 0.
omitemptywas added to this field in the same change that introduced theseCEL rules (4.17 → 4.18). Branches
release-4.18throughrelease-4.22allcarry both;
release-4.17has neither and usesjson:"currentRevision"without
omitempty.Related prior fixes for the same class of problem:
self >= oldSelf/ "must only increase"Note that
library-go'snode_controller.goapplies nodeStatus via SSA withapplyoperatorv1.NodeStatus().WithNodeName(...)and nocurrentRevisionfield, so whichever field manager wins ownership can drop the field entirely.
That looks like a plausible mechanism for reaching the
0/unset state, thoughwe did not confirm it directly.
Reproduction
The trigger in our case was that etcd static pods could not start, because the
masters could not reach the bootstrap etcd member on 2379/2380 (our own
security group misconfiguration — not an OKD bug). But the operator's response
to that condition is the issue being reported: rather than surfacing the
underlying fault, it wedges itself on a validation error it can never satisfy.
on 2379/2380 is one way)
cluster-etcd-operatorbegins emitting the CEL error continuouslyWorkaround
Removing the CEL validations lets the operator write a status where
currentRevisionis genuinely unset:oc patch crd etcds.operator.openshift.io --type=json \ -p='[{"op":"replace","path":"/spec/versions/0/schema/openAPIV3Schema/properties/status/properties/nodeStatuses/items/x-kubernetes-validations","value":[]}]'CVO reverts the CRD within about 45 seconds, but that window is enough. After
the patch, rejections dropped from ~150/min to 0 and stayed at 0 even once
CVO restored the rules — because the stored state was then unset rather than
0, which satisfies the transition rule.Outcome
After fixing the underlying network issue and applying the CRD patch, the
cluster completed successfully. All three nodes are now at
currentRevision: 8with zero rejections.
We cannot say with certainty whether the CEL deadlock would have cleared on its
own once the network was fixed, or whether the patch was required. What is
clear is that while etcd pods could not start, the operator was stuck in a loop
it could not exit, and the validation error completely obscured the real cause.
Suggested fix
Either:
omitemptyfromCurrentRevision int32so0serialises explicitlyand
has()behaves as the CEL rules assume, or0/unset ambiguity, e.g.!has(oldSelf.currentRevision) || oldSelf.currentRevision == 0 || has(self.currentRevision)Option 1 matches the pre-4.18 behaviour.
Full operator logs, the
EtcdCR, and the CRD as captured during thedeadlock are available on request.