.NET: fix(dotnet): persist workflow scope aliases consistently

docs/decisions/0048-dotnet-declarative-state-ownership.md

@@ -0,0 +1,77 @@
+---
+status: proposed
+contact: baywet
+date: 2026-10-28
+deciders: westey-m
+---
+
+# Explicit ownership for .NET declarative Local state
+
+## Context and Problem Statement
+
+Declarative `Local` and `Topic` variables share canonical `Local` storage and a sensitivity sidecar. Older
+checkpoints can instead contain declarative values in physical `Topic` storage. Code-first executors can also use
+physical `Topic` independently, including the same key as a declarative variable.
+
+Sensitivity metadata alone cannot identify which physical value it describes. A preflight check cannot prevent
+this ambiguity when different fan-out executors queue raw `Topic` and canonical `Local` writes in the same
+superstep: each executor sees only its own pending updates.
+
+## Decision Drivers
+
+- Preserve unrelated code-first state and the native representation of canonical values.
+- Continue reading and selectively migrating older checkpoints without guessing assignment order.
+- Keep declarative storage conventions out of the language-neutral workflow state manager.
+- Avoid full-scope scans for individual reads and writes.
+- Preserve read-only expression snapshots and the existing checkpoint restoration publication boundary.
+
+## Considered Options
+
+### Infer ownership from sensitivity metadata and visible keys
+
+Good: no additional persisted metadata or core API changes.
+Bad: executor-private pending updates defeat the preflight, and committed sensitivity metadata cannot distinguish
+canonical ownership from an older declarative `Topic` value.
+
+### Validate combined updates before publication
+
+Good: a validator can inspect updates from every executor before they are committed.
+Bad: this needs a new language-neutral validation extension point or declarative scope knowledge in core
+publication. Rejecting every same-key pair would also prevent independently owned raw and canonical values.
+
+### Persist versioned canonical ownership separately
+
+Good: plain helpers, wrapped contexts, and restoration can distinguish ownership after concurrent publication,
+without changing native values or public workflow APIs.
+Bad: writes and checkpoints include one additional private entry per canonical Local key. Older runtimes do not
+understand that ownership when both physical scopes contain the same key.
+
+## Decision Outcome
+
+Chosen option: "Persist versioned canonical ownership separately", because the ambiguity belongs to declarative
+storage interpretation and does not require changing core publication semantics.
+
+For each new canonical `Local` assignment or migrated legacy entry, persist integer version `1` under the same key
+in the private `__Microsoft_Agents_AI_Workflows_Declarative_Ownership:Local` scope, alongside the existing value and
+sensitivity sidecar. No ownership entry is needed for other scopes.
+
+A recognized ownership entry means the sensitivity sidecar describes canonical `Local` state. Any same-key raw
+`Topic` value remains independent and must not be read as a legacy value, migrated, or deleted by declarative
+helpers or restoration. This permits mixed fan-out writes to coexist. Unsupported ownership versions are rejected
+when consulted to disambiguate a `Topic` entry.
+
+Without ownership metadata, preserve the existing conservative compatibility rules: a metadata-marked `Topic` key
+without a `Local` value is legacy; overlapping marked values are rejected before migration writes; unmarked raw
+`Topic` entries remain untouched. Visible unmarked collisions still reject new canonical assignments unless
+canonical ownership is already recorded. Missing ownership alone does not trigger a write during restoration.
+
+Migration records canonical ownership before removing only the migrated legacy key. Plain helper null updates
+physically delete the canonical value, sensitivity, and ownership entries. Wrapped declarative clears retain the
+existing stored blank representation with its sensitivity and ownership.
+
+Point operations use key-level `PortableValue` existence probes, including for metadata with an explicit `None`
+value. Scope enumeration remains appropriate for whole-state restoration and read-only snapshots, but not for
+each point operation in a growing batch. Ad hoc expression snapshots never persist migration or ownership updates.
+
+Checkpoint stores must preserve these private entries together with workflow values. This format is
+backward-readable by the new runtime; resuming mixed-scope checkpoints on older runtimes is not supported.

dotnet/src/Microsoft.Agents.AI.Workflows.Declarative/Extensions/PortableValueExtensions.cs

@@ -14,26 +14,32 @@ namespace Microsoft.Agents.AI.Workflows.Declarative.Extensions;
 internal static class PortableValueExtensions
 {
     public static FormulaValue ToFormula(this PortableValue value) =>
+        value.ToPortableObject() switch
+        {
+            Dictionary<string, PortableValue> recordValue => recordValue.ToRecord(),
+            IDictionary recordValue => recordValue.ToRecord(),
+            PortableValue[] tableValue => tableValue.ToTable(),
+            var nativeValue => nativeValue.ToFormula(),
+        };
+
+    public static object ToPortableObject(this PortableValue value) =>
         value.TypeId switch
         {
-            null => FormulaValue.NewBlank(),
-            _ when value.TypeId.IsMatch<UnassignedValue>() => FormulaValue.NewBlank(),
-            _ when value.IsType(out string? stringValue) => FormulaValue.New(stringValue),
-            _ when value.IsSystemType(out bool? boolValue) => FormulaValue.New(boolValue.Value),
-            _ when value.IsSystemType(out int? intValue) => FormulaValue.New(intValue.Value),
-            _ when value.IsSystemType(out long? longValue) => FormulaValue.New(longValue.Value),
-            _ when value.IsSystemType(out decimal? decimalValue) => FormulaValue.New(decimalValue.Value),
-            _ when value.IsSystemType(out float? floatValue) => FormulaValue.New(floatValue.Value),
-            _ when value.IsSystemType(out double? doubleValue) => FormulaValue.New(doubleValue.Value),
-            _ when value.IsParentType(out Dictionary<string, PortableValue>? recordValue) => recordValue.ToRecord(),
-            _ when value.IsParentType(out IDictionary? recordValue) => recordValue.ToRecord(),
-            _ when value.IsType(out PortableValue[]? tableValue) => tableValue.ToTable(),
-            _ when value.IsType(out ChatMessage? messageValue) => messageValue.ToRecord(),
-            _ when value.IsType(out DateTime dateValue) =>
-                dateValue.TimeOfDay == TimeSpan.Zero ?
-                    FormulaValue.NewDateOnly(dateValue.Date) :
-                    FormulaValue.New(dateValue),
-            _ when value.IsType(out TimeSpan timeValue) => FormulaValue.New(timeValue),
+            null => UnassignedValue.Instance,
+            _ when value.TypeId.IsMatch<UnassignedValue>() => UnassignedValue.Instance,
+            _ when value.IsType(out string? stringValue) => stringValue,
+            _ when value.IsSystemType(out bool? boolValue) => boolValue.Value,
+            _ when value.IsSystemType(out int? intValue) => intValue.Value,
+            _ when value.IsSystemType(out long? longValue) => longValue.Value,
+            _ when value.IsSystemType(out decimal? decimalValue) => decimalValue.Value,
+            _ when value.IsSystemType(out float? floatValue) => floatValue.Value,
+            _ when value.IsSystemType(out double? doubleValue) => doubleValue.Value,
+            _ when value.IsParentType(out Dictionary<string, PortableValue>? recordValue) => recordValue,
+            _ when value.IsParentType(out IDictionary? recordValue) => recordValue,
+            _ when value.IsType(out PortableValue[]? tableValue) => tableValue,
+            _ when value.IsType(out ChatMessage? messageValue) => messageValue,
+            _ when value.IsType(out DateTime dateValue) => dateValue,
+            _ when value.IsType(out TimeSpan timeValue) => timeValue,
             _ => throw new DeclarativeModelException($"Unsupported portable type: {value.TypeId.TypeName}"),
         };
 

dotnet/src/Microsoft.Agents.AI.Workflows.Declarative/Interpreter/DeclarativeActionExecutor.cs

@@ -40,6 +40,7 @@ protected DeclarativeActionExecutor(DialogAction model, WorkflowFormulaState sta
         }
 
         this.State = state;
+        this.RegisterCheckpointRestoringCallback(state, state.RestoreCheckpointAsync);
 
         this.Model = model;
     }
@@ -125,13 +126,6 @@ public override async ValueTask HandleAsync(ActionExecutorResult message, IWorkf
 
     protected abstract ValueTask<object?> ExecuteAsync(IWorkflowContext context, CancellationToken cancellationToken = default);
 
-    /// <summary>
-    /// Restore the state of the executor from a checkpoint.
-    /// This must be overridden to restore any state that was saved during checkpointing.
-    /// </summary>
-    protected override ValueTask OnCheckpointRestoredAsync(IWorkflowContext context, CancellationToken cancellationToken = default) =>
-        this.State.RestoreAsync(context, cancellationToken);
-
     protected async ValueTask AssignAsync(PropertyPath? targetPath, FormulaValue result, IWorkflowContext context, SensitivityLevel sensitivity = SensitivityLevel.None)
     {
         if (targetPath is null)

dotnet/src/Microsoft.Agents.AI.Workflows.Declarative/Interpreter/DeclarativeWorkflowContext.cs

@@ -93,6 +93,8 @@ internal async ValueTask QueueStateUpdateAsync<T>(
     /// <inheritdoc/>
     public async ValueTask<TValue?> ReadStateAsync<TValue>(string key, string? scopeName = null, CancellationToken cancellationToken = default)
     {
+        scopeName = IsManagedScope(scopeName) ? WorkflowFormulaState.GetScopeName(scopeName) : scopeName;
+
         return typeof(TValue) switch
         {
             // Not a managed scope, just pass through.  This is valid when a declarative
@@ -107,6 +109,8 @@ _ when typeof(TValue) == typeof(FormulaValue) => (TValue?)(object?)this.State.Ge
 
     public async ValueTask<TValue> ReadOrInitStateAsync<TValue>(string key, Func<TValue> initialStateFactory, string? scopeName = null, CancellationToken cancellationToken = default)
     {
+        scopeName = IsManagedScope(scopeName) ? WorkflowFormulaState.GetScopeName(scopeName) : scopeName;
+
         return typeof(TValue) switch
         {
             // Not a managed scope, just pass through.  This is valid when a declarative
@@ -115,40 +119,86 @@ public async ValueTask<TValue> ReadOrInitStateAsync<TValue>(string key, Func<TVa
             // Retrieve formula values directly from the managed state to avoid conversion.
             _ when typeof(TValue) == typeof(FormulaValue) => await EnsureFormulaValueAsync().ConfigureAwait(false),
             // Retrieve native types from the source context to avoid conversion.
-            _ => await this.Source.ReadOrInitStateAsync(key, initialStateFactory, scopeName, cancellationToken).ConfigureAwait(false),
+            _ => await EnsureNativeValueAsync().ConfigureAwait(false),
         };
 
+        async ValueTask<TValue> EnsureNativeValueAsync()
+        {
+            PortableValue? existing = await this.Source.ReadStateAsync<PortableValue>(key, scopeName, cancellationToken).ConfigureAwait(false);
+            bool legacyEntry = existing is null &&
+                await Kit.IWorkflowContextExtensions.HasLegacyTopicEntryAsync(this.Source, key, scopeName!, forWrite: true, cancellationToken).ConfigureAwait(false);
+            bool initialized = false;
+            TValue result = await this.Source.ReadOrInitStateAsync(key, () =>
+            {
+                initialized = true;
+                return initialStateFactory();
+            }, legacyEntry ? VariableScopeNames.Topic : scopeName, cancellationToken).ConfigureAwait(false);
+            if (legacyEntry)
+            {
+                PortableValue legacyValue = await ReadLegacyValueAsync().ConfigureAwait(false);
+                await this.Source.QueueStateUpdateAsync(key, legacyValue.ToPortableObject(), scopeName, cancellationToken).ConfigureAwait(false);
+            }
+            if (initialized || legacyEntry)
+            {
+                await CompleteInitializationAsync(legacyEntry).ConfigureAwait(false);
+            }
+            return result;
+        }
+
         async ValueTask<TValue> EnsureFormulaValueAsync()
         {
             Debug.Assert(typeof(TValue) == typeof(FormulaValue), "It is a bug to call this method with TValue not === FormulaValue");
             FormulaValue? result = this.State.Get(key, scopeName);
 
             if (result is null or BlankValue)
             {
-                result = initialStateFactory() as FormulaValue;
+                bool legacyEntry = await Kit.IWorkflowContextExtensions.HasLegacyTopicEntryAsync(this.Source, key, scopeName!, forWrite: true, cancellationToken).ConfigureAwait(false);
+                PortableValue? legacyValue = legacyEntry ? await ReadLegacyValueAsync().ConfigureAwait(false) : null;
+                result = legacyValue is null ? initialStateFactory() as FormulaValue : legacyValue.ToFormula();
                 if (result is null)
                 {
                     throw new InvalidOperationException($"The initial state factory for key '{key}' in scope '{scopeName}' did not return a FormulaValue.");
                 }
 
                 this.State.Set(key, result, scopeName);
-                await this.Source.QueueStateUpdateAsync(key, result.AsPortable(), scopeName, cancellationToken)
+                await this.Source.QueueStateUpdateAsync(key, legacyValue is null ? result.AsPortable() : legacyValue.ToPortableObject(), scopeName, cancellationToken)
                                  .ConfigureAwait(false);
+                await CompleteInitializationAsync(legacyEntry).ConfigureAwait(false);
             }
 
             return (TValue)(object)result!; // The null analyzer is confused here, but it is impossible to hit this line with result is null
         }
+
+        async ValueTask<PortableValue> ReadLegacyValueAsync() =>
+            await this.Source.ReadStateAsync<PortableValue>(key, VariableScopeNames.Topic, cancellationToken).ConfigureAwait(false)
+                ?? throw new InvalidOperationException($"The existing state for key '{key}' in scope '{VariableScopeNames.Topic}' is missing.");
+
+        async ValueTask CompleteInitializationAsync(bool legacyEntry)
+        {
+            SensitivityLevel sensitivity = legacyEntry
+                ? await this.Source.ReadStateAsync<SensitivityLevel>(key, WorkflowFormulaState.GetSensitivityScopeName(scopeName!), cancellationToken).ConfigureAwait(false)
+                : SensitivityLevel.None;
+            await this.QueueSensitivityUpdateAsync(key, scopeName, sensitivity, cancellationToken).ConfigureAwait(false);
+            this.State.SetSensitivity(key, scopeName, sensitivity);
+            if (legacyEntry)
+            {
+                await this.Source.QueueStateUpdateAsync<object>(key, null, VariableScopeNames.Topic, cancellationToken).ConfigureAwait(false);
+            }
+        }
     }
 
     /// <inheritdoc/>
     public ValueTask<HashSet<string>> ReadStateKeysAsync(string? scopeName = null, CancellationToken cancellationToken = default)
-        => this.Source.ReadStateKeysAsync(scopeName, cancellationToken);
+    {
+        scopeName = IsManagedScope(scopeName) ? WorkflowFormulaState.GetScopeName(scopeName) : scopeName;
+        return this.Source.ReadStateKeysAsync(scopeName, cancellationToken);
+    }
 
     /// <inheritdoc/>
     public ValueTask SendMessageAsync(object message, string? targetId = null, CancellationToken cancellationToken = default)
         => this.Source.SendMessageAsync(message, targetId, cancellationToken);
 
-    public ValueTask UpdateStateAsync<T>(
+    public async ValueTask UpdateStateAsync<T>(
         string key,
         T? value,
         string? scopeName,
@@ -164,23 +214,39 @@ scopeName is not null && // null scope cannot be managed
         {
             // Not a managed scope, just pass through.  This is valid when a declarative
             // workflow has been ejected to code (where DeclarativeWorkflowContext is also utilized).
-            return this.Source.QueueStateUpdateAsync(key, value, scopeName, cancellationToken);
+            await this.Source.QueueStateUpdateAsync(key, value, scopeName, cancellationToken).ConfigureAwait(false);
+            return;
         }
 
         if (!ManagedScopes.Contains(scopeName!) && !allowSystem)
         {
             throw new DeclarativeActionException($"Cannot manage variable definitions in scope: '{scopeName}'.");
         }
 
-        return value switch
+        scopeName = WorkflowFormulaState.GetScopeName(scopeName);
+
+        bool legacyEntry = await Kit.IWorkflowContextExtensions.HasLegacyTopicEntryAsync(this.Source, key, scopeName, forWrite: true, cancellationToken).ConfigureAwait(false);
+        switch (value)
         {
-            null => QueueEmptyStateAsync(),
-            UnassignedValue => QueueEmptyStateAsync(),
-            BlankValue => QueueEmptyStateAsync(),
-            FormulaValue formulaValue => QueueFormulaStateAsync(formulaValue),
-            DataValue dataValue => QueueDataValueStateAsync(dataValue),
-            _ => QueueNativeStateAsync(value),
-        };
+            case null:
+            case UnassignedValue:
+            case BlankValue:
+                await QueueEmptyStateAsync().ConfigureAwait(false);
+                break;
+            case FormulaValue formulaValue:
+                await QueueFormulaStateAsync(formulaValue).ConfigureAwait(false);
+                break;
+            case DataValue dataValue:
+                await QueueDataValueStateAsync(dataValue).ConfigureAwait(false);
+                break;
+            default:
+                await QueueNativeStateAsync(value).ConfigureAwait(false);
+                break;
+        }
+        if (legacyEntry)
+        {
+            await this.Source.QueueStateUpdateAsync<object>(key, null, VariableScopeNames.Topic, cancellationToken).ConfigureAwait(false);
+        }
 
         async ValueTask QueueEmptyStateAsync()
         {
@@ -237,6 +303,6 @@ private ValueTask QueueSensitivityUpdateAsync(string key, string? scopeName, Sen
             return default;
         }
 
-        return this.Source.QueueStateUpdateAsync(key, sensitivity, WorkflowFormulaState.GetSensitivityScopeName(scopeName), cancellationToken);
+        return WorkflowFormulaState.QueueSensitivityUpdateAsync(this.Source, key, scopeName, sensitivity, delete: false, cancellationToken);
     }
 }