Skip to content

feat(alloy): ship Kubernetes events to in-cluster Loki - #381

Open
stevenolen wants to merge 2 commits into
mainfrom
alloy-k8s-events
Open

stevenolen wants to merge 2 commits into
mainfrom
alloy-k8s-events

Conversation

@stevenolen

Copy link
Copy Markdown
Contributor

Description

Kubernetes events expire from the API server after about an hour, so they are gone by the time we debug an incident after the fact (for example a Connect content-pod ImagePullBackOff investigated two days later: pod logs were in Loki, events were not). This adds a loki.source.kubernetes_events component to the shared Alloy config so events from all namespaces are retained in each workload's in-cluster Loki, labeled with cluster like the existing pod and Karpenter log sources.

Code Flow

  • buildAlloyConfig (lib/steps/helm_helpers.go) now emits loki.source.kubernetes_events "events", which forwards to a loki.relabel "kubernetes_events" that adds the cluster label and forwards to loki.write.local. The generator is shared, so both the AWS (awsHelmAlloy) and Azure (azureHelmAlloy) helm steps pick it up.
  • Query with {job="loki.source.kubernetes_events"}; the namespace label is the involved object's namespace.
  • All namespaces are watched because the events most useful for postmortems (image pulls, scheduling, Karpenter, node and CNI issues) are often outside posit-team.
  • RBAC: the alloy chart's ClusterRole already grants get/list/watch on core events; no change needed.
  • Duplication: Alloy runs as a DaemonSet, so each replica ships the same events. On our Alloy version (chart 0.12.6, Alloy v1.7.5) this component has no clustering support (added in Alloy v1.16.0). The copies are byte-identical (same labels, event timestamp and line), and Loki deduplicates identical entries at ingest and query time. The existing loki.source.kubernetes "pods" source works the same way today.
  • Not yet validated by a real Alloy binary: run ptd ensure <staging-target> --only-steps helm --dry-run, apply, and confirm Alloy pods stay healthy and {job="loki.source.kubernetes_events"} returns results before rolling out widely.

Category of change

  • Bug fix (non-breaking change which fixes an issue)
  • Version upgrade (upgrading the version of a service or product)
  • New feature (non-breaking change which adds functionality)
  • Build: a code change that affects the build system or external dependencies
  • Performance: a code change that improves performance
  • Refactor: a code change that neither fixes a bug nor adds a feature
  • Documentation: documentation changes
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist

  • I have reviewed my own diff and added inline comments on lines I want reviewers to focus on or that I am uncertain about

Kubernetes events expire from the API server after about an hour, so they
are gone when debugging an incident after the fact. Add a
loki.source.kubernetes_events component to the shared Alloy config so
events from all namespaces are retained in each workload's Loki, labeled
with cluster like the existing pod and Karpenter sources.
@stevenolen
stevenolen marked this pull request as ready for review November 20, 2026 18:20
@stevenolen
stevenolen requested a review from a team as a code owner November 20, 2026 18:20
@claude

claude Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Claude finished @stevenolen's task in 1m 42s —— View job


Code Review

  • Gather context
  • Read changed files
  • Review guidelines
  • Post review

The change is small, correct, and follows existing patterns well. One issue in the test, one observation on the config design.

lib/steps/helm_helpers_test.go — potential panic

The strings.Index / substring slice on line 39 runs regardless of whether the earlier assert.Contains on line 25 passed. If the loki.source.kubernetes_events block is absent (e.g., after a template refactor), start is -1 and config[start:start+end] panics rather than failing the test cleanly. Use require.Contains on lines 25 and 28 so the test stops on failure before reaching the index arithmetic. I've left an inline comment with a suggested fix.

lib/steps/helm_helpers.go — loki.source.kubernetes vs. loki.source.kubernetes_events wiring

The two existing log sources (pods, karpenter) go directly to loki.write.local.receiver without a relabel hop; the cluster label is added in the upstream discovery.relabel block. For the new events source there is no discovery.relabel, so a separate loki.relabel step is the right call here — the asymmetry is justified, not a consistency problem. Worth a brief comment alongside the relabel block noting why (no __meta_* labels available from loki.source.kubernetes_events to relabel from a discovery stage), so a future reader doesn't wonder if the hop was an oversight.

Minor

The PR description calls out that the RBAC is already covered by the chart's existing ClusterRole — good. No concern there.

Overall: looks good to merge once the require fix is in.

}`)

// Events are collected cluster-wide, so no namespaces restriction is set.
start := strings.Index(config, `loki.source.kubernetes_events "events"`)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the loki.source.kubernetes_events "events" component is ever absent, the preceding assert.Contains (line 25) won't stop execution, so config[start:] here will panic (start == -1). Use require.Contains on lines 25 and 28 so the test fails cleanly instead of panicking.

Suggested change
start := strings.Index(config, `loki.source.kubernetes_events "events"`)
require.Contains(t, config, `loki.source.kubernetes_events "events" {
forward_to = [loki.relabel.kubernetes_events.receiver]
}`)
require.Contains(t, config, `loki.relabel "kubernetes_events" {
forward_to = [loki.write.local.receiver]
rule {
action = "replace"
target_label = "cluster"
replacement = "test-cluster-name"
}
}`)
// Events are collected cluster-wide, so no namespaces restriction is set.
start := strings.Index(config, `loki.source.kubernetes_events "events"`)

You'd also need to update the import: require is github.com/stretchr/testify/require.

@stevenolen

Copy link
Copy Markdown
Contributor Author

@claude again

@claude

claude Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Claude finished @stevenolen's task in 16s —— View job


Tasks

  • Gather context
  • Read test file — fix already applied
  • No changes needed

The require.Contains fix from my previous review is already in place — lines 26 and 29 of lib/steps/helm_helpers_test.go already use require.Contains, and the require import is present on line 8. Nothing to do here.

@stevenolen
stevenolen enabled auto-merge November 20, 2026 20:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant