Repository navigation
feat(alloy): ship Kubernetes events to in-cluster Loki - #381
stevenolen wants to merge 2 commits into
Conversation
Kubernetes events expire from the API server after about an hour, so they are gone when debugging an incident after the fact. Add a loki.source.kubernetes_events component to the shared Alloy config so events from all namespaces are retained in each workload's Loki, labeled with cluster like the existing pod and Karpenter sources.
|
Claude finished @stevenolen's task in 1m 42s —— View job Code Review
The change is small, correct, and follows existing patterns well. One issue in the test, one observation on the config design.
|
| }`) | ||
|
|
||
| // Events are collected cluster-wide, so no namespaces restriction is set. | ||
| start := strings.Index(config, `loki.source.kubernetes_events "events"`) |
There was a problem hiding this comment.
If the loki.source.kubernetes_events "events" component is ever absent, the preceding assert.Contains (line 25) won't stop execution, so config[start:] here will panic (start == -1). Use require.Contains on lines 25 and 28 so the test fails cleanly instead of panicking.
| start := strings.Index(config, `loki.source.kubernetes_events "events"`) | |
| require.Contains(t, config, `loki.source.kubernetes_events "events" { | |
| forward_to = [loki.relabel.kubernetes_events.receiver] | |
| }`) | |
| require.Contains(t, config, `loki.relabel "kubernetes_events" { | |
| forward_to = [loki.write.local.receiver] | |
| rule { | |
| action = "replace" | |
| target_label = "cluster" | |
| replacement = "test-cluster-name" | |
| } | |
| }`) | |
| // Events are collected cluster-wide, so no namespaces restriction is set. | |
| start := strings.Index(config, `loki.source.kubernetes_events "events"`) |
You'd also need to update the import: require is github.com/stretchr/testify/require.
|
@claude again |
|
Claude finished @stevenolen's task in 16s —— View job Tasks
The |
Description
Kubernetes events expire from the API server after about an hour, so they are gone by the time we debug an incident after the fact (for example a Connect content-pod ImagePullBackOff investigated two days later: pod logs were in Loki, events were not). This adds a
loki.source.kubernetes_eventscomponent to the shared Alloy config so events from all namespaces are retained in each workload's in-cluster Loki, labeled withclusterlike the existing pod and Karpenter log sources.Code Flow
buildAlloyConfig(lib/steps/helm_helpers.go) now emitsloki.source.kubernetes_events "events", which forwards to aloki.relabel "kubernetes_events"that adds theclusterlabel and forwards toloki.write.local. The generator is shared, so both the AWS (awsHelmAlloy) and Azure (azureHelmAlloy) helm steps pick it up.{job="loki.source.kubernetes_events"}; thenamespacelabel is the involved object's namespace.events; no change needed.loki.source.kubernetes "pods"source works the same way today.ptd ensure <staging-target> --only-steps helm --dry-run, apply, and confirm Alloy pods stay healthy and{job="loki.source.kubernetes_events"}returns results before rolling out widely.Category of change
Checklist