fix(observe): unify daemon logs through single channel - #605
Open
mccormickt wants to merge 2 commits into
Open
mccormickt wants to merge 2 commits into
mccormickt wants to merge 2 commits into
Conversation
dmah42
reviewed
Sep 12, 2026
|
|
||
| // Clear cached readiness after draining the buffer so the | ||
| // next call to `readable_mut` waits for more events. | ||
| if events.read == 0 { |
Contributor
There was a problem hiding this comment.
part of a different change?
Contributor
Author
There was a problem hiding this comment.
Happy to split it out if you'd like, but this was another fix that I encountered and is required for this change to work as well. I missed this in #601, and not clearing readiness caused the Tokio worker pool to be consumed by log reader tasks. This would block any gRPC startup tasks from making progress and caused a livelock.
dmah42
reviewed
Sep 12, 2026
dmah42
previously approved these changes
Sep 12, 2026
mccormickt
force-pushed
the
push-wzqwsnpsqxxx
branch
from
September 12, 2026 16:07
2b3d655 to
64f583c
Compare
Auraed polls each per-CPU eBPF perf buffer through Tokio AsyncFd. After a buffer became readable, the reader drained it but left AsyncFd readiness cached. Empty reads then completed immediately and the reader tasks consumed the Tokio worker pool. This prevented gRPC startup, cell requests, and signal handling from making progress. Clear readiness after an empty read so the task waits for the next kernel notification. Keep readiness set after a non-empty read so busy buffers continue draining.
Use one LogChannel for the tracing subscriber and ObserveService. Stop log forwarders during graceful shutdown.
mccormickt
force-pushed
the
push-wzqwsnpsqxxx
branch
from
September 12, 2026 16:17
64f583c to
e9eaa6f
Compare
dmah42
approved these changes
Sep 12, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR unifies auraed's logging patterns through a single
LogChannel, leveragingtracing_subscriber::MakeWriterand a newBroadcastWriterstruct to properly broadcast log messages to all subscribers.Before this change,
aer observe get-aurae-daemon-log-streamwould never display logs.Now:
We also now properly handle shutdown of log forwarders, so an
ObserveServiceclient does not prevent auraed from exiting and shuts down properly.