Skip to content

Python: Isolate declarative MCP client sessions by workflow - #8662

Open
Vincent Biret (baywet) wants to merge 79 commits into
mainfrom
fix/mcp-workflow-session-isolation
Open

Vincent Biret (baywet) wants to merge 79 commits into
mainfrom
fix/mcp-workflow-session-isolation

Conversation

@baywet

Copy link
Copy Markdown
Member

Motivation & Context

The default declarative MCP handlers cached connected clients by endpoint, label, connection name, and headers. When one handler instance served multiple workflow sessions, those sessions could therefore reuse the same stateful MCP protocol session. This change fixes the MCP client session isolation/cache scoping bug while preserving session continuity within one workflow session and the existing endpoint, header, label, and connection isolation.

Description & Review Guide

  • What are the major changes? Added framework-owned workflow session identity to the Python invocation contract and a workflow-scoped .NET handler contract, propagated trusted session IDs from workflow execution contexts, and included that identity in default MCP client cache keys. Added regression coverage for separate workflows, same-workflow continuation, cache-key discrimination, and concurrent cache creation.
  • What is the impact of these changes? Separate workflow sessions no longer share a stateful MCP protocol session through a shared default handler. Calls within the same workflow session continue to reuse the cached MCP client when all existing cache identity dimensions also match. Custom handlers using the existing .NET contract remain compatible.
  • What do you want reviewers to focus on? Please review the workflow-owned scope propagation and the compatibility behavior for custom handlers and direct unscoped handler calls.

Related Issue

No public issue is available for this MCP client session isolation/cache scoping bug.

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR is linked to an issue and there is no other open PR for this issue (see Related Issue above).
  • This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 7ed91283-cb21-49a7-98f4-b1c088d0b6b6
Copilot AI balanced review requested due to automatic review settings September 22, 2026 18:08
@agent-framework-automation agent-framework-automation Bot added documentation Usage: [Issues, PRs], Target: documentation in the code base and learn docs python Usage: [Issues, PRs], Target: Python .NET Usage: [Issues, PRs], Target: .Net labels Sep 22, 2026
@agent-framework-automation agent-framework-automation Bot added the workflows Usage: [Issues, PRs], Target: Workflows label Sep 22, 2026
@github-actions github-actions Bot changed the title .NET/Python: Isolate declarative MCP client sessions by workflow .NET: .NET/Python: Isolate declarative MCP client sessions by workflow Sep 22, 2026
@github-actions github-actions Bot changed the title .NET: .NET/Python: Isolate declarative MCP client sessions by workflow Python: .NET/Python: Isolate declarative MCP client sessions by workflow Sep 22, 2026

This comment was marked as outdated.

@github-code-quality

github-code-quality Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Code coverage overview

Languages: Python

Python / code-coverage/python

main 6afcecd fix/mcp-workflow-ses... 7e45ede +/-
92% 92% 0%
View coverage of the 10 most impacted files.
File main 6afcecd fix/mcp-workflow-ses... 7e45ede +/-
packages/declar..._mcp_handler.py 98% 96% -2%
packages/declar...cutors_tools.py 99% 98% -1%
packages/declar...arative_base.py 93% 93% 0%
packages/declar...tive_builder.py 99% 99% 0%
packages/core/a...flows/_agent.py 92% 92% 0%
packages/core/a...ws/_workflow.py 94% 94% 0%
packages/core/a...lows/_runner.py 98% 99% +1%
packages/declar...utors_agents.py 91% 93% +2%
packages/core/a...low_executor.py 87% 89% +2%
packages/core/a...uest_context.py 0% 85% +85%

ℹ️ Calculations represent line coverage. Results may change as calculations are updated.


Updated October 27, 2026 23:55 UTC

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MAF Automated Review — Iteration 1

Result: Findings reported
Scope: full PR (1 commit(s)): bb9e92b189ed
Model: gpt-5.6-sol-fast

Overview

The change adds workflow identity to both default MCP cache keys while preserving the existing endpoint, label, connection, and header dimensions, and its regression tests cover normal isolation, continuation, and same-key creation coalescing. Provider-backed calls also retain their per-invocation lifecycle and shutdown guards. Two residual gaps remain: the .NET compatibility fallback is scoped to a reusable workflow object rather than an execution session, and Python's completed-entry LRU does not bound the now-per-workflow set of concurrent connection attempts.

Reviewed the supplied pull-request change set across correctness, security/reliability, architecture, and failure behavior.
2 verified findings remained after source verification (2 medium) across 2 files. Details are attached to the affected lines below.

Affected areas: dotnet/src/Microsoft.Agents.AI.Workflows.Declarative/Interpreter/DeclarativeWorkflowContext.cs, python/packages/declarative/agent_framework_declarative/_workflows/_mcp_handler.py

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment thread python/packages/declarative/tests/test_default_mcp_tool_handler.py Fixed
Comment thread python/packages/declarative/tests/test_default_mcp_tool_handler.py Fixed

This comment was marked as outdated.

@baywet

Copy link
Copy Markdown
Member Author

Addressed the latest review summary item about .NET global client lock contention in current branch changes: cache misses now reserve/coalesce an in-flight creation under _clientLock, run CreateClientAsync outside the global lock, and publish/evict under the lock afterward. Added NoProvider_ConcurrentWorkflowSessionCreations_DoNotSerializeHandshakeAsync to cover concurrent workflow handshakes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 89548cb9-9583-47c1-a00f-675aebf810f8
Use per-invocation MCP sessions when the workflow scope is missing or empty, and reject empty scopes at the cache boundary. Cover isolation, real SDK lifecycle, cancellation, and shutdown while preserving scoped cache reuse.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8e076c3e-5f45-4049-9cf5-f51ed5971ba8
@baywet

Copy link
Copy Markdown
Member Author

I checked the Git history for the dependency-floor item. WORKFLOW_AGENT_RUN_KEY was introduced on this PR branch in b10b5c4 (whose then-current package metadata said Core 1.20.0), and the initial-state initializer hook was introduced in 839e4f0 (then Core 1.22.0). Neither commit is contained in any Python release tag, including the latest python-1.23.0. The current head now declares Core >=1.23.0,<2 after merging main, but that released Core still lacks these branch-only hooks. Consequently, the package versions recorded when those commits were authored are not valid release floors, and I cannot identify a released minimum that includes the integration. Eduard van Valkenburg (@eavanvalkenburg) how would you suggest handling this: coordinate a new Core release floor with the release owners, or preserve the existing range through a compatibility adapter? I am addressing the runner-context compatibility item separately and will leave this review open pending your guidance.

Make queued-response inspection an optional capability and adapt legacy contexts through their existing non-persisting checkpoint snapshots. Preserve queued-response validation without draining messages or consuming pending requests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8e076c3e-5f45-4049-9cf5-f51ed5971ba8
@baywet

Copy link
Copy Markdown
Member Author

Addressed item 2 of #8662 (review) in cdedd69. get_queued_response_info_events is no longer required by the exported RunnerContext protocol; it is an optional inspection capability. Legacy contexts are adapted through their existing non-persisting build_checkpoint API, so validation still considers already queued responses without draining messages or consuming pending requests. Three regressions failed with AttributeError before the fix and now pass: streaming/non-streaming legacy continuations and rejection of a combined queued/submitted batch without losing requests, followed by a successful retry. Validation: docker exec -w /workspace/python af-8662-review-48ae /tmp/review-venv/bin/python -m pytest packages/core/tests/workflow packages/declarative/tests -q -o addopts= -o cache_dir=/tmp/af-8662-core-cache — passed (3,315 passed, 11 skipped, 1 expected failure). Ruff lint/format, strict source Pyright, and all five configured test type checkers passed. Dependency metadata is unchanged; item 1 remains pending your guidance in #8662 (comment). Leaving the review open as requested.

Comment thread python/packages/core/agent_framework/_workflows/_runner_context.py
Assert the concrete context used by the nested checkpoint test before accessing its optional queue-inspection capability.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8e076c3e-5f45-4049-9cf5-f51ed5971ba8
@baywet

Copy link
Copy Markdown
Member Author

Fixed the latest Test Typing Checks failure in c7bfe99: the nested checkpoint regression now asserts InProcRunnerContext before accessing its optional queue-inspection capability. The original Pyright attribute error reproduced locally; all five configured test type checkers now pass, along with Ruff lint/format and all 16 affected checkpoint test variants. Requested a Copilot review of this head. The existing human review and dependency-floor discussion remain open.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-cutting session isolation and recursive checkpoint recovery require maintainer validation, and the retained restore-identity defect remains unresolved.

1 open issue

Review effort: Balanced

Comment thread python/packages/core/agent_framework/_workflows/_agent.py Outdated
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8e076c3e-5f45-4049-9cf5-f51ed5971ba8

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-cutting session ownership, nested checkpoint migration, and cancellation-sensitive cleanup require maintainer validation beyond static review.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Propagate agent context across nested workflow execution

python/​packages/​declarative/​agent_framework_declarative/​_workflows/​_declarative_builder.py:149

When a declarative child is wrapped in WorkflowExecutor and the parent runs through as_agent(), process_workflow() calls child.run(...) without forwarding the parent's agent context. Two turns with the same AgentSession therefore reach this initializer with is_agent_run=False and receive different MCP scopes, losing cached session continuity. Propagate the trusted continuation context across the subworkflow boundary while preserving isolation between child bindings, and add a nested same-session regression test.

1 resolved since last review

Review effort: Balanced

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8e076c3e-5f45-4049-9cf5-f51ed5971ba8
@baywet

Copy link
Copy Markdown
Member Author

Addressed the nested agent-context finding from #8662 (review) in a9fe090. WorkflowExecutor now forwards the trusted parent agent-run context after a fresh child run is accepted and before execution. Response-only continuations and checkpoint restores are unchanged. Each binding still owns a unique child workflow instance and its own MCP namespace. The 16 new regression variants cover streaming/non-streaming, text/message-list inputs, explicit/sessionless agent turns, and one/two nesting levels, with two sibling bindings. All failed the same-session scope-reuse assertion before the fix and now pass; they also verify distinct scopes for sibling bindings, different AgentSessions, and subsequent direct runs. Validation: docker exec -e PYTHONPATH=/workspace/python/packages/core:/workspace/python/packages/declarative:/workspace/python/packages/openai -w /workspace/python af-8662-review-48ae /tmp/review-venv/bin/python -m pytest packages/core/tests/workflow packages/declarative/tests -q -o addopts= -o cache_dir=/tmp/af-8662-nested-cache — passed (3,339 passed, 11 skipped, 1 expected failure). Ruff lint/format, strict source Pyright, and all five configured test type checkers passed. Existing human discussions remain open.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8e076c3e-5f45-4049-9cf5-f51ed5971ba8
@baywet

Copy link
Copy Markdown
Member Author

Fixed the latest Pre-commit Hooks failure in 0d767ce. Adding postponed annotations made the existing quoted Workflow annotation redundant; the pinned pyupgrade hook required it to be unquoted. Applied exactly that one-line correction. pyupgrade 3.21.2 (--py310-plus), Ruff lint/format, strict source Pyright, and 18 focused nested-workflow tests pass. Merge-gatekeeper failed solely because of that hook failure; the new head will rerun normal CI.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Nested invocation and parent-entry replay paths still leave gaps in MCP session isolation and continuity.

1 open issue
1 previously missed issue

In code that hasn't changed since last review

Medium severity Propagate parent scope to workflow-backed agents

python/​packages/​declarative/​agent_framework_declarative/​_workflows/​_mcp_handler.py:102

When InvokeAzureAgent invokes a registered declarative workflow.as_agent() without an explicit session, unrelated parent sessions reuse the same child sessionless scope and cached MCP connection. _invoke_agent_and_store_results() does not supply parent identity by default, and the new WorkflowExecutor propagation does not cover this path. Propagate the parent workflow scope to workflow-backed agents while preserving top-level sessionless continuation, and add a regression with two parent sessions invoking the same child agent.

Review effort: Balanced

Comment thread python/packages/core/agent_framework/_workflows/_workflow_executor.py Outdated
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8e076c3e-5f45-4049-9cf5-f51ed5971ba8
@baywet

Copy link
Copy Markdown
Member Author

Investigated the separate workflow-backed-agent finding in #8662 (review). The reported InvokeAzureAgent -> registered workflow.as_agent() path does not currently reach child execution: both streaming and non-streaming parent regressions fail with TypeError: WorkflowAgent.run() got an unexpected keyword argument 'options', wrapped as AgentInvalidResponseException. _invoke_agent_and_store_results always passes options, but WorkflowAgent.run does not accept it; no child MCP invocation occurs. Consequently I have not reproduced cross-parent MCP reuse through this path and am not claiming it fixed. Per the agreed scope, I left that integration unchanged rather than enabling a currently unsupported path as part of this fix. Please clarify the supported invocation/adapter path intended by this finding, or whether enabling that integration should be separate work. Leaving this summary item open for guidance. The independently reproduced parent-entry replay finding is addressed in 7b957d1.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Resumed child requests leave ancestor scopes unchanged, allowing downstream MCP calls to reuse another run’s session.

1 open issue
1 resolved since last review

Review effort: Balanced

Comment thread python/packages/core/agent_framework/_workflows/_workflow.py
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 8e076c3e-5f45-4049-9cf5-f51ed5971ba8

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-cutting workflow scope and checkpoint changes alter core continuation semantics, requiring maintainer validation of stored-checkpoint compatibility, nested restoration, and cancellation behavior beyond isolated verification.

No open issues 🎉

2 previously missed issues

In code that hasn't changed since last review

Medium severity Preserve parent request origins across deferred and replayed responses

python/​packages/​core/​agent_framework/​_workflows/​_workflow.py:473

Ordinary parent ctx.request_info() requests do not save or restore their run origin; only forwarded child requests populate the origin map. If parent runs A and B pause before a declarative child, answering B and then A makes both child invocations derive B's scope and reuse its MCP session. Capture origins for ordinary core requests too, include them in mixed-scope batch validation, and restore them before response handlers execute, including checkpoint replay.

Medium severity Avoid deep-copying non-migrated request payloads

python/​packages/​core/​agent_framework/​_workflows/​_workflow.py:1205

This unconditionally deep-copies every original request payload, even in a core workflow with no declarative migration policy. An in-process request containing a threading.Lock can be emitted without checkpointing, but answering it now fails with TypeError: cannot pickle '_thread.lock' object before its response handler runs. Avoid copying payloads for read-only validation, and restrict migration snapshots to the metadata that migration actually changes.

1 resolved since last review

Review effort: Balanced

Capture ordinary request origins and restore them before response handlers and cancellation. Validate response trees without copying arbitrary request payloads, and defer metadata migration until validation succeeds.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8e076c3e-5f45-4049-9cf5-f51ed5971ba8
@baywet

Copy link
Copy Markdown
Member Author

Addressed both summary-only findings from #8662 (review) in e1b150d.

  • Ordinary Core requests now capture their originating framework context, participate in mixed-origin validation, and restore it before response/cancellation handlers execute. Nested boundaries use the same bookkeeping. Regression cases cover B-then-A responses, streaming/non-streaming execution, rebuilt file checkpoints, and queued-response replay before invoking a declarative child.
  • Response-tree validation no longer deep-copies live request payloads. Validators run read-only first; missing declarative scope metadata is updated only after every validator succeeds. Non-checkpointed requests containing a threading.Lock retain the original dictionary and lock identity through response handling. Existing checkpoint/migration atomicity coverage remains passing.

All eight new regression cases failed for the reported behaviors before the implementation and now pass. Container validation:

docker exec -e PYTHONPATH=/workspace/python/packages/core:/workspace/python/packages/declarative:/workspace/python/packages/openai -w /workspace/python af-8662-review-48ae /tmp/review-venv/bin/python -m pytest packages/core/tests packages/declarative/tests -q -o addopts= -o cache_dir=/tmp/af-8662-core-request-cache --tb=short --disable-warnings

Result: 10,681 passed, 193 skipped, 1 xfailed (exit 0). Ruff lint/format, pinned pyupgrade, source Pyright, and all five required test type checkers also passed. Request-origin documentation was updated alongside the implementation.

The existing maintainer-guidance discussions remain open; this batch does not change the dependency floor or enable the unsupported workflow-backed-agent call path.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical validation gaps in cancellation preflight and wrapped-reply batches can mix response origins and execute downstream work under the wrong restored scope.

2 open issues

Review effort: Balanced

Comment thread python/packages/core/agent_framework/_workflows/_workflow.py
Comment thread python/packages/core/agent_framework/_workflows/_workflow.py Outdated
Validate graph-routed wrapped replies before response consumption and each live dispatch batch. Include queued replies in cancellation preflight while preserving optional runner-context inspection and deferred metadata updates.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8e076c3e-5f45-4049-9cf5-f51ed5971ba8
Comment thread python/packages/core/tests/workflow/test_sub_workflow.py

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Manually composed Core workflows retain a stale declarative scope, causing fresh runs to reuse cached stateful MCP clients and child workflows to ignore inherited parent-scope changes.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

Use the active Core workflow scope for declarative executor lookups while preserving factory and standalone initialization. Cover fresh runs, nested bindings, agent continuity, entry replay, and interleaved approvals.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 8e076c3e-5f45-4049-9cf5-f51ed5971ba8
@baywet

Copy link
Copy Markdown
Member Author

Addressed the manual-composition concern in Copilot review 5675089233 in 7e45ede. Declarative executors manually composed with Core WorkflowBuilder now use the active Core scope, including inherited child scopes, rather than a stale declarative cache. Factory scope initialization and standalone state-helper behavior are preserved.

Test-first evidence: all ten new parameterized cases failed on unchanged 16b1655 because separate fresh runs reused a scope, then passed with the fix. Coverage includes direct/nested streaming and non-streaming runs, explicit/sessionless agent continuity, rebuilt entry-checkpoint replay, and interleaved approval continuations with live/checkpoint-restored state.

Validation passed: full containerized Core/declarative pytest suites (10,701 passed, 193 skipped, 1 xfailed), Ruff lint/format, pinned pyupgrade, strict source Pyright, and all five repository-configured test type checkers. The existing OpenTelemetry shutdown exporter warning remains non-fatal. The separate “statement has no effect” finding on await run was answered and resolved with execution evidence; that await is required to drive the lazy non-streaming continuation.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because same-ID request relays can break MCP session continuity, while repeated origin-map copying introduces quadratic event-loop work during large fan-outs.

No open issues 🎉

Review effort: Balanced

This branch was successfully deployed

1 active deployment
github-app-auth — 7e45edea Deployed Oct 27, 2026 by baywet via add_label #26370
integration — e5886f4f Deployed Oct 19, 2026 by baywet via dotnet-foundry-hosted-it #30716
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Usage: [Issues, PRs], Target: documentation in the code base and learn docs .NET Usage: [Issues, PRs], Target: .Net python Usage: [Issues, PRs], Target: Python workflows Usage: [Issues, PRs], Target: Workflows

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants