☂️ Improve health checks based on node conditions

[prashanth26]

Context

Currently there are node conditions added to the node by different actors like kubelet , Node Problem Detector(NPD), Network Problem Detector. But MCM acts only on a few by default and not that effectively.

Customer could update the conditions on shoot level from this section.
Current shoot defaults are

- ReadonlyFilesystem(NPD)
- KernelDeadlock (NPD)
- DiskPressure(kubelet)

MCM has its own defaults but ofcourse they are overridden by shoot defaults (or values provided by customer)
MCM defaults are:

- KernelDeadlock (NPD)
- ReadonlyFilesystem (NPD)
- DiskPressure (kubelet)
- NetworkUnavailable (kubelet)

Example of conditions added by Network problem detector are

  - lastHeartbeatTime: "2023-02-22T07:27:28Z"
    lastTransitionTime: "2023-02-20T14:18:15Z"
    message: no cluster network problems
    reason: NoNetworkProblems
    status: "False"
    type: ClusterNetworkProblem
  - lastHeartbeatTime: "2023-02-22T07:27:23Z"
    lastTransitionTime: "2023-02-21T13:59:48Z"
    message: no host network problems
    reason: NoNetworkProblems
    type: HostNetworkProblem
    status: "False"

Goal

MCM should use the node conditions more effectively and do the replacement if it feels the node is unhealthy according to the condition.

Quick Improvements

• Identify Unrecoverable nodeConditions (ex- ReadonlyFilesystem) and don't wait healthTimeout, but immediately make machine Failed.
  • could be deceiving as it is reported if other file system is read-only Gauge for FilesystemIsReadOnly not downgraded to 0 after fixing the problem kubernetes/node-problem-detector#474
  • taint based evictions don’t follow PDB (confirm, if yes , then do health timeout 0) , Documentation on taint-based pod evictions implies respect of PDBs, which may not be the case kubernetes/website#7829
• MCM should consider node Conditions added by Network Problem Detector also.

Research

• collect metrics on currently acted upon node Conditions
• relevance of KernelDeadlock condition. is it just for docker ? gardener doesn't support docker . Is it a permanent condition ?
• look into effectiveness of taint node by condition . Refer similar issue on NPD Add option flag to taint node for permanent problems kubernetes/node-problem-detector#457 . Currently the tainting is done by KCM only, and only for kubelet added node conditions
• feasibility of firing remediation process of NPD
• edit NPD config to introduce new node conditions which could be beneficial for Gardener scenario
• feasibility of different timeouts for different conditions , or would it be to much fine tuning. see live issue # 2653

Why is this needed:

To have better , reliable recovery of nodes and lesser downtimes for customer workloads.