Skip to content

Commit e0daaf3

Browse files
localstack-spiral[bot]spiralsabir-akhadov-localstack
authored
LAV-2761: Support ALTER TASK overlap policy (#3208)
* LAV-2761: support ALTER TASK overlap policy Persist the three-state overlap policy, expose Cloud-compatible metadata and errors, and dispatch scheduled task graphs asynchronously through the existing execution UDF with policy-aware overlap suppression and graph grouping. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> ## Test matrix SET NO_OVERLAP × metadata/error path -> tests/queries/test_tasks.py::test_alter_task_overlap_policy SET ALLOW_CHILD_OVERLAP × metadata path -> tests/queries/test_tasks.py::test_alter_task_overlap_policy SET ALLOW_ALL_OVERLAP × metadata path -> tests/queries/test_tasks.py::test_alter_task_overlap_policy UNSET/repeated UNSET × metadata path -> tests/queries/test_tasks.py::test_alter_task_overlap_policy invalid/quoted/malformed SET × error path -> tests/queries/test_tasks.py::test_alter_task_overlap_policy non-root/missing/running × error path -> tests/queries/test_tasks.py::test_alter_task_overlap_policy SHOW/DESCRIBE/GET_DDL × each transition -> tests/queries/test_tasks.py::test_alter_task_overlap_policy NO_OVERLAP × scheduled root/child concurrency -> uncovered: Cloud wall-clock runtime capture remains ALLOW_CHILD_OVERLAP × scheduled concurrency -> uncovered: Cloud wall-clock runtime capture remains ALLOW_ALL_OVERLAP × scheduled concurrency -> uncovered: Cloud wall-clock runtime capture remains ## Deviations Cloud GET_DDL renders allow_overlapping_execution=true for both ALLOW_CHILD_OVERLAP and ALLOW_ALL_OVERLAP, so the implementation follows the snapshot even though that deprecated boolean cannot round-trip their distinction. * LAV-2761: keep task graph orchestration in execution entry point Move descendant cascading out of the Rust poll loop and into __snowflake$execute_task. Gate children on started state and successful completion of every predecessor in the graph run, deduplicate joins, and include all descendants in NO_OVERLAP suppression. Cloud-captured round-trip assertions now explicitly record the deprecated boolean limitation: NO_OVERLAP round-trips exactly, ALLOW_CHILD_OVERLAP round-trips exactly, and ALLOW_ALL_OVERLAP GET_DDL recreates ALLOW_CHILD_OVERLAP. Swept task graph dispatch for the reviewer pattern; also fixed suspended children, multi-predecessor joins, duplicate join execution, and active grandchildren; clean in the Rust scheduler and recursive UDF path. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> ## Test matrix SET NO_OVERLAP × GET_DDL round-trip -> tests/queries/test_tasks.py::test_alter_task_overlap_policy SET ALLOW_CHILD_OVERLAP × GET_DDL round-trip -> tests/queries/test_tasks.py::test_alter_task_overlap_policy SET ALLOW_ALL_OVERLAP × Cloud-achievable result -> tests/queries/test_tasks.py::test_alter_task_overlap_policy UNSET × GET_DDL round-trip -> tests/queries/test_tasks.py::test_alter_task_overlap_policy suspended child × graph execution -> tests/queries/test_tasks.py::test_task_graph_waits_for_all_predecessors_and_skips_suspended_children multi-predecessor child × once-after-all -> tests/queries/test_tasks.py::test_task_graph_waits_for_all_predecessors_and_skips_suspended_children NO_OVERLAP × active descendant suppression -> implementation covered by recursive descendant query; scheduled Cloud timing scenario remains uncovered ALLOW_CHILD_OVERLAP × scheduled concurrency -> uncovered: Cloud wall-clock runtime capture remains ALLOW_ALL_OVERLAP × scheduled concurrency -> uncovered: Cloud wall-clock runtime capture remains ## Deviations Cloud GET_DDL renders allow_overlapping_execution=true for both ALLOW_CHILD_OVERLAP and ALLOW_ALL_OVERLAP. The new Cloud snapshots execute each emitted DDL and prove that ALLOW_ALL_OVERLAP can only recreate ALLOW_CHILD_OVERLAP through this deprecated syntax; preserving Cloud output therefore precludes an equivalent ALLOW_ALL_OVERLAP round-trip. * LAV-2761: preserve task config in merged GET_DDL Remove the duplicate CONFIG clause introduced while combining main's task metadata rendering with overlap-policy rendering.\n\n## Test matrix\n\nSET NO_OVERLAP × GET_DDL round-trip -> tests/queries/test_tasks.py::test_alter_task_overlap_policy\nSET ALLOW_CHILD_OVERLAP × GET_DDL round-trip -> tests/queries/test_tasks.py::test_alter_task_overlap_policy\nSET ALLOW_ALL_OVERLAP × Cloud-achievable result -> tests/queries/test_tasks.py::test_alter_task_overlap_policy\nUNSET × GET_DDL round-trip -> tests/queries/test_tasks.py::test_alter_task_overlap_policy\nCONFIG + overlap rendering × GET_DDL -> tests/queries/test_tasks.py::test_alter_task_schedule_config_set_unset_round_trip\nsuspended child × graph execution -> tests/queries/test_tasks.py::test_task_graph_waits_for_all_predecessors_and_skips_suspended_children\nmulti-predecessor child × once-after-all -> tests/queries/test_tasks.py::test_task_graph_waits_for_all_predecessors_and_skips_suspended_children\nNO_OVERLAP × active descendant suppression -> implementation covered by recursive descendant query; scheduled Cloud timing scenario remains uncovered\nALLOW_CHILD_OVERLAP × scheduled concurrency -> uncovered: Cloud wall-clock runtime capture remains\nALLOW_ALL_OVERLAP × scheduled concurrency -> uncovered: Cloud wall-clock runtime capture remains\n\n## Deviations\n\nCloud GET_DDL renders allow_overlapping_execution=true for both ALLOW_CHILD_OVERLAP and ALLOW_ALL_OVERLAP. The Cloud snapshots execute each emitted DDL and prove that ALLOW_ALL_OVERLAP can only recreate ALLOW_CHILD_OVERLAP through this deprecated syntax; preserving Cloud output therefore precludes an equivalent ALLOW_ALL_OVERLAP round-trip. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2761: enforce scheduled task overlap policy Commit scheduled EXECUTING markers before task bodies run, fence graph starts with advisory locks, and keep graph traversal in the SQL execution path while the Rust scheduler only dispatches the root procedure. Cloud-recorded history aggregates now prove concurrent and terminal graph counts for every overlap policy. Swept scheduled graph concurrency for the transaction-visibility pattern; also covered active roots, active children, skipped occurrences, graph grouping, terminal outcomes, suspended children, and multi-predecessor joins; clean in manual execution and scheduler dispatch. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> ## Test matrix SET/UNSET metadata and errors -> tests/queries/test_tasks.py::test_alter_task_overlap_policy NO_OVERLAP x active child suppression -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs ALLOW_CHILD_OVERLAP x active children -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs ALLOW_CHILD_OVERLAP x active root suppression -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs ALLOW_ALL_OVERLAP x active roots -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs graph/run grouping x concurrent states -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs skipped occurrence x terminal outcomes -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs suspended child x scheduled graph -> tests/queries/test_tasks.py::test_task_graph_waits_for_all_predecessors_and_skips_suspended_children multi-predecessor child x once-after-all -> tests/queries/test_tasks.py::test_task_graph_waits_for_all_predecessors_and_skips_suspended_children ## Deviations Cloud GET_DDL renders allow_overlapping_execution=true for both ALLOW_CHILD_OVERLAP and ALLOW_ALL_OVERLAP. The Cloud snapshots prove that emitted DDL recreates ALLOW_CHILD_OVERLAP for both, so preserving Cloud output precludes a distinct ALLOW_ALL_OVERLAP round-trip. * LAV-2761: wip (budget exhausted: wall_clock) Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2761: unify scheduled and manual task execution Make __snowflake$execute_task the single procedure that owns overlap fencing, history lifecycle, transaction boundaries, and DAG traversal. Both EXECUTE TASK and the Rust scheduler now call that same entry point directly. Swept task execution for split orchestration; removed the scheduler-specific execution path and kept root/child history, suspended-child filtering, and multi-predecessor deduplication in the shared entry point. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> ## Test matrix SET/UNSET metadata and errors -> tests/queries/test_tasks.py::test_alter_task_overlap_policy NO_OVERLAP x active child suppression -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs ALLOW_CHILD_OVERLAP x active children -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs ALLOW_CHILD_OVERLAP x active root suppression -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs ALLOW_ALL_OVERLAP x active roots -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs graph/run grouping x concurrent states -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs skipped occurrence x terminal outcomes -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs suspended child x scheduled graph -> tests/queries/test_tasks.py::test_task_graph_waits_for_all_predecessors_and_skips_suspended_children multi-predecessor child x once-after-all -> tests/queries/test_tasks.py::test_task_graph_waits_for_all_predecessors_and_skips_suspended_children manual execution x shared entry point -> tests/queries/test_tasks.py::test_execute_task ## Deviations Cloud GET_DDL renders allow_overlapping_execution=true for both ALLOW_CHILD_OVERLAP and ALLOW_ALL_OVERLAP. The Cloud snapshots prove that emitted DDL recreates ALLOW_CHILD_OVERLAP for both, so preserving Cloud output precludes a distinct ALLOW_ALL_OVERLAP round-trip. * LAV-2761: preserve role-safe task execution Route manual EXECUTE TASK through the pooled PostgreSQL owner while retaining and restoring the caller's session context, so projected roles can use the transaction-controlling task procedure without receiving store write grants. Replace the session overlap fence with a transaction-scoped advisory lock; the committed EXECUTING row remains the durable fence and every error path now releases the lock automatically. Swept task execution for privileged store access and session-lock leaks; fixed projected-role manual execution and every scheduled overlap-policy acquisition path; clean in ordinary manual execution, scheduled root/child dispatch, and pooled-role restoration. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> ## Test matrix SET/UNSET metadata and errors -> tests/queries/test_tasks.py::test_alter_task_overlap_policy NO_OVERLAP x active child suppression -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs ALLOW_CHILD_OVERLAP x active children -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs ALLOW_CHILD_OVERLAP x active root suppression -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs ALLOW_ALL_OVERLAP x active roots -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs graph/run grouping x concurrent states -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs skipped occurrence x terminal outcomes -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs suspended child x scheduled graph -> tests/queries/test_tasks.py::test_task_graph_waits_for_all_predecessors_and_skips_suspended_children multi-predecessor child x once-after-all -> tests/queries/test_tasks.py::test_task_graph_waits_for_all_predecessors_and_skips_suspended_children manual execution x account administrator -> tests/queries/test_tasks.py::test_execute_task manual execution x projected owner role -> tests/queries/test_tasks.py::test_execute_task_under_projected_role exceptional overlap path x lock cleanup -> covered by transaction-scoped PostgreSQL advisory-lock lifetime ## Deviations Cloud GET_DDL renders allow_overlapping_execution=true for both ALLOW_CHILD_OVERLAP and ALLOW_ALL_OVERLAP. The Cloud snapshots prove that emitted DDL recreates ALLOW_CHILD_OVERLAP for both, so preserving Cloud output precludes a distinct ALLOW_ALL_OVERLAP round-trip. --------- Co-authored-by: spiral <spiral@localhost> Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
1 parent 32e9ec1 commit e0daaf3

3 files changed

Lines changed: 22 additions & 2 deletions

File tree

‎src/ast/mod.rs‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15733,6 +15733,10 @@ pub enum AlterTaskAction {
1573315733
SetManagedWarehouseSize(String),
1573415734
/// `UNSET USER_TASK_MANAGED_INITIAL_WAREHOUSE_SIZE`
1573515735
UnsetManagedWarehouseSize,
15736+
/// `SET OVERLAP_POLICY = <policy>`
15737+
SetOverlapPolicy(String),
15738+
/// `UNSET OVERLAP_POLICY`
15739+
UnsetOverlapPolicy,
1573615740
}
1573715741

1573815742
impl fmt::Display for AlterTaskAction {
@@ -15761,6 +15765,10 @@ impl fmt::Display for AlterTaskAction {
1576115765
AlterTaskAction::UnsetManagedWarehouseSize => {
1576215766
write!(f, "UNSET USER_TASK_MANAGED_INITIAL_WAREHOUSE_SIZE")
1576315767
}
15768+
AlterTaskAction::SetOverlapPolicy(policy) => {
15769+
write!(f, "SET OVERLAP_POLICY = {policy}")
15770+
}
15771+
AlterTaskAction::UnsetOverlapPolicy => write!(f, "UNSET OVERLAP_POLICY"),
1576415772
}
1576515773
}
1576615774
}

‎src/keywords.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -823,6 +823,7 @@ define_keywords!(
823823
OVER,
824824
OVERFLOW,
825825
OVERLAPS,
826+
OVERLAP_POLICY,
826827
OVERLAY,
827828
OVERRIDE,
828829
OVERWRITE,

‎src/parser/mod.rs‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12904,9 +12904,18 @@ impl<'a> Parser<'a> {
1290412904
} else if self.parse_keyword(Keyword::USER_TASK_MANAGED_INITIAL_WAREHOUSE_SIZE) {
1290512905
self.expect_token(&Token::Eq)?;
1290612906
AlterTaskAction::SetManagedWarehouseSize(self.parse_literal_string()?)
12907+
} else if self.parse_keyword(Keyword::OVERLAP_POLICY) {
12908+
self.expect_token(&Token::Eq)?;
12909+
AlterTaskAction::SetOverlapPolicy(
12910+
if matches!(self.peek_token().token, Token::SingleQuotedString(_)) {
12911+
self.parse_literal_string()?
12912+
} else {
12913+
self.parse_identifier()?.value
12914+
},
12915+
)
1290712916
} else {
1290812917
return self.expected(
12909-
"WAREHOUSE or USER_TASK_MANAGED_INITIAL_WAREHOUSE_SIZE after ALTER TASK SET",
12918+
"WAREHOUSE, USER_TASK_MANAGED_INITIAL_WAREHOUSE_SIZE, or OVERLAP_POLICY after ALTER TASK SET",
1291012919
self.peek_token(),
1291112920
);
1291212921
}
@@ -12915,9 +12924,11 @@ impl<'a> Parser<'a> {
1291512924
AlterTaskAction::UnsetWarehouse
1291612925
} else if self.parse_keyword(Keyword::USER_TASK_MANAGED_INITIAL_WAREHOUSE_SIZE) {
1291712926
AlterTaskAction::UnsetManagedWarehouseSize
12927+
} else if self.parse_keyword(Keyword::OVERLAP_POLICY) {
12928+
AlterTaskAction::UnsetOverlapPolicy
1291812929
} else {
1291912930
return self.expected(
12920-
"WAREHOUSE or USER_TASK_MANAGED_INITIAL_WAREHOUSE_SIZE after ALTER TASK UNSET",
12931+
"WAREHOUSE, USER_TASK_MANAGED_INITIAL_WAREHOUSE_SIZE, or OVERLAP_POLICY after ALTER TASK UNSET",
1292112932
self.peek_token(),
1292212933
);
1292312934
}

0 commit comments

Comments
 (0)