Scheduler with WorkGraph slice - #7625
Draft
agoscinski wants to merge 5 commits into
Draft
agoscinski wants to merge 5 commits into
agoscinski wants to merge 5 commits into
Conversation
Route broker tasks through named persistent queues so submission can be
separated from execution. The default queue keeps the legacy 'tasks'
storage directory and behaviour; additional queues get 'tasks-<name>'
directories with validated names. Queue membership is tracked per
subscription, dispatch only serves subscribed workers, and ACKs route to
the owning queue with a fallback scan that also covers post-restart
recovery. 'queue' threads from task messages and subscriptions through
the communicator, 'LoopCommunicator', 'RemoteProcessThreadController'
and 'Runner', defaulting to current behaviour everywhere. Covered by
queue-routing unit tests; existing broker suites stay green.
✨ Add scheduler service gating submissions
Provide the 'aiida.scheduler' package with a 'Scheduler' service that
subscribes to the scheduler queue and forwards admitted tasks to the
worker queue. It reuses the controller submission vocabulary and creates
its communicator behind 'create_communicator' for later transports.
Covered by unit tests, live forwarding tests, and a real process
submitted through the scheduler queue to a daemon worker.
♻️ Compose controller in scheduler service
Give the scheduler an owned 'RemoteProcessThreadController' for the
forwarding path instead of inheriting it. A scheduler uses submission
vocabulary, it is not a controller: composition keeps the public surface
narrow and controller evolution from leaking into the scheduler as
orchestration grows. No behaviour change.
✨ Track task lifecycle from completion broadcasts
Teach the scheduler the return leg of the loop: it subscribes to worker
completion broadcasts ('scheduler.task.completed/failed') and records
them in a lifecycle registry with in-flight counts. Unknown subjects are
ignored; completions for unknown or terminal tasks count as duplicates,
never as state changes. Worker-side announcements are the follow-up; the
receiving half and its subjects are settled here.
rm linbreak ✨ Detect process kinds for scheduler throttling Classify submissions without touching the message protocol or engine submission path: launch bodies resolve their process-class identifier through an explicit registry with loader fallback, continue bodies load the checkpoint node by pid and read node_type. Kinds drive the per-kind concurrency caps. Document the message-handling logic as pseudocode in the module docstring. 🔧 Drop legacy task-id aliases from scheduler Completions resolve by stamped scheduler id only. The alias map translating submitter-provided task ids was backward-compat glue for a protocol nobody speaks; the duplicate counter already signals genuinely unknown completions. ✨ Add WorkGraphScheduler orchestration interface Provide an abstract workgraph policy owned as a scheduler member: workgraph-kind records delegate initial and follow-up dispatches to it while containers themselves are never forwarded. The 'workgraph' kind is reserved and mapped ahead of the future node type; without an installed member such tasks take the normal path. Returned bodies re-enter core admission. ✨ Announce worker completions via ProcessListener Add a plumpy ProcessListener bound per launched task that publishes terminal receipts to the durable completions queue. No Process subclass changes: listeners are additive. Wiring into launchers is documented but left unwired. Rename scheduler event listener ♻️ Remove scheduler workgraph orchestration
scheduler + workgraph This reverts commit 4c5e8e0038100b3e1ff03435bf2df93647c14173. ✨ Classify scheduler workgraph tasks
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #7625 +/- ##
===========================================
- Coverage 83.34% 33.16% -50.17%
===========================================
Files 614 617 +3
Lines 50774 51064 +290
===========================================
- Hits 42312 16932 -25380
- Misses 8462 34132 +25670 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.