Commit e0daaf3
LAV-2761: Support ALTER TASK overlap policy (#3208)
* LAV-2761: support ALTER TASK overlap policy
Persist the three-state overlap policy, expose Cloud-compatible metadata and errors, and dispatch scheduled task graphs asynchronously through the existing execution UDF with policy-aware overlap suppression and graph grouping.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
## Test matrix
SET NO_OVERLAP × metadata/error path -> tests/queries/test_tasks.py::test_alter_task_overlap_policy
SET ALLOW_CHILD_OVERLAP × metadata path -> tests/queries/test_tasks.py::test_alter_task_overlap_policy
SET ALLOW_ALL_OVERLAP × metadata path -> tests/queries/test_tasks.py::test_alter_task_overlap_policy
UNSET/repeated UNSET × metadata path -> tests/queries/test_tasks.py::test_alter_task_overlap_policy
invalid/quoted/malformed SET × error path -> tests/queries/test_tasks.py::test_alter_task_overlap_policy
non-root/missing/running × error path -> tests/queries/test_tasks.py::test_alter_task_overlap_policy
SHOW/DESCRIBE/GET_DDL × each transition -> tests/queries/test_tasks.py::test_alter_task_overlap_policy
NO_OVERLAP × scheduled root/child concurrency -> uncovered: Cloud wall-clock runtime capture remains
ALLOW_CHILD_OVERLAP × scheduled concurrency -> uncovered: Cloud wall-clock runtime capture remains
ALLOW_ALL_OVERLAP × scheduled concurrency -> uncovered: Cloud wall-clock runtime capture remains
## Deviations
Cloud GET_DDL renders allow_overlapping_execution=true for both ALLOW_CHILD_OVERLAP and ALLOW_ALL_OVERLAP, so the implementation follows the snapshot even though that deprecated boolean cannot round-trip their distinction.
* LAV-2761: keep task graph orchestration in execution entry point
Move descendant cascading out of the Rust poll loop and into __snowflake$execute_task. Gate children on started state and successful completion of every predecessor in the graph run, deduplicate joins, and include all descendants in NO_OVERLAP suppression.
Cloud-captured round-trip assertions now explicitly record the deprecated boolean limitation: NO_OVERLAP round-trips exactly, ALLOW_CHILD_OVERLAP round-trips exactly, and ALLOW_ALL_OVERLAP GET_DDL recreates ALLOW_CHILD_OVERLAP.
Swept task graph dispatch for the reviewer pattern; also fixed suspended children, multi-predecessor joins, duplicate join execution, and active grandchildren; clean in the Rust scheduler and recursive UDF path.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
## Test matrix
SET NO_OVERLAP × GET_DDL round-trip -> tests/queries/test_tasks.py::test_alter_task_overlap_policy
SET ALLOW_CHILD_OVERLAP × GET_DDL round-trip -> tests/queries/test_tasks.py::test_alter_task_overlap_policy
SET ALLOW_ALL_OVERLAP × Cloud-achievable result -> tests/queries/test_tasks.py::test_alter_task_overlap_policy
UNSET × GET_DDL round-trip -> tests/queries/test_tasks.py::test_alter_task_overlap_policy
suspended child × graph execution -> tests/queries/test_tasks.py::test_task_graph_waits_for_all_predecessors_and_skips_suspended_children
multi-predecessor child × once-after-all -> tests/queries/test_tasks.py::test_task_graph_waits_for_all_predecessors_and_skips_suspended_children
NO_OVERLAP × active descendant suppression -> implementation covered by recursive descendant query; scheduled Cloud timing scenario remains uncovered
ALLOW_CHILD_OVERLAP × scheduled concurrency -> uncovered: Cloud wall-clock runtime capture remains
ALLOW_ALL_OVERLAP × scheduled concurrency -> uncovered: Cloud wall-clock runtime capture remains
## Deviations
Cloud GET_DDL renders allow_overlapping_execution=true for both ALLOW_CHILD_OVERLAP and ALLOW_ALL_OVERLAP. The new Cloud snapshots execute each emitted DDL and prove that ALLOW_ALL_OVERLAP can only recreate ALLOW_CHILD_OVERLAP through this deprecated syntax; preserving Cloud output therefore precludes an equivalent ALLOW_ALL_OVERLAP round-trip.
* LAV-2761: preserve task config in merged GET_DDL
Remove the duplicate CONFIG clause introduced while combining main's task metadata rendering with overlap-policy rendering.\n\n## Test matrix\n\nSET NO_OVERLAP × GET_DDL round-trip -> tests/queries/test_tasks.py::test_alter_task_overlap_policy\nSET ALLOW_CHILD_OVERLAP × GET_DDL round-trip -> tests/queries/test_tasks.py::test_alter_task_overlap_policy\nSET ALLOW_ALL_OVERLAP × Cloud-achievable result -> tests/queries/test_tasks.py::test_alter_task_overlap_policy\nUNSET × GET_DDL round-trip -> tests/queries/test_tasks.py::test_alter_task_overlap_policy\nCONFIG + overlap rendering × GET_DDL -> tests/queries/test_tasks.py::test_alter_task_schedule_config_set_unset_round_trip\nsuspended child × graph execution -> tests/queries/test_tasks.py::test_task_graph_waits_for_all_predecessors_and_skips_suspended_children\nmulti-predecessor child × once-after-all -> tests/queries/test_tasks.py::test_task_graph_waits_for_all_predecessors_and_skips_suspended_children\nNO_OVERLAP × active descendant suppression -> implementation covered by recursive descendant query; scheduled Cloud timing scenario remains uncovered\nALLOW_CHILD_OVERLAP × scheduled concurrency -> uncovered: Cloud wall-clock runtime capture remains\nALLOW_ALL_OVERLAP × scheduled concurrency -> uncovered: Cloud wall-clock runtime capture remains\n\n## Deviations\n\nCloud GET_DDL renders allow_overlapping_execution=true for both ALLOW_CHILD_OVERLAP and ALLOW_ALL_OVERLAP. The Cloud snapshots execute each emitted DDL and prove that ALLOW_ALL_OVERLAP can only recreate ALLOW_CHILD_OVERLAP through this deprecated syntax; preserving Cloud output therefore precludes an equivalent ALLOW_ALL_OVERLAP round-trip.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2761: enforce scheduled task overlap policy
Commit scheduled EXECUTING markers before task bodies run, fence graph starts with advisory locks, and keep graph traversal in the SQL execution path while the Rust scheduler only dispatches the root procedure. Cloud-recorded history aggregates now prove concurrent and terminal graph counts for every overlap policy.
Swept scheduled graph concurrency for the transaction-visibility pattern; also covered active roots, active children, skipped occurrences, graph grouping, terminal outcomes, suspended children, and multi-predecessor joins; clean in manual execution and scheduler dispatch.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
## Test matrix
SET/UNSET metadata and errors -> tests/queries/test_tasks.py::test_alter_task_overlap_policy
NO_OVERLAP x active child suppression -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs
ALLOW_CHILD_OVERLAP x active children -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs
ALLOW_CHILD_OVERLAP x active root suppression -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs
ALLOW_ALL_OVERLAP x active roots -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs
graph/run grouping x concurrent states -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs
skipped occurrence x terminal outcomes -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs
suspended child x scheduled graph -> tests/queries/test_tasks.py::test_task_graph_waits_for_all_predecessors_and_skips_suspended_children
multi-predecessor child x once-after-all -> tests/queries/test_tasks.py::test_task_graph_waits_for_all_predecessors_and_skips_suspended_children
## Deviations
Cloud GET_DDL renders allow_overlapping_execution=true for both ALLOW_CHILD_OVERLAP and ALLOW_ALL_OVERLAP. The Cloud snapshots prove that emitted DDL recreates ALLOW_CHILD_OVERLAP for both, so preserving Cloud output precludes a distinct ALLOW_ALL_OVERLAP round-trip.
* LAV-2761: wip (budget exhausted: wall_clock)
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2761: unify scheduled and manual task execution
Make __snowflake$execute_task the single procedure that owns overlap fencing, history lifecycle, transaction boundaries, and DAG traversal. Both EXECUTE TASK and the Rust scheduler now call that same entry point directly.
Swept task execution for split orchestration; removed the scheduler-specific execution path and kept root/child history, suspended-child filtering, and multi-predecessor deduplication in the shared entry point.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
## Test matrix
SET/UNSET metadata and errors -> tests/queries/test_tasks.py::test_alter_task_overlap_policy
NO_OVERLAP x active child suppression -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs
ALLOW_CHILD_OVERLAP x active children -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs
ALLOW_CHILD_OVERLAP x active root suppression -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs
ALLOW_ALL_OVERLAP x active roots -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs
graph/run grouping x concurrent states -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs
skipped occurrence x terminal outcomes -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs
suspended child x scheduled graph -> tests/queries/test_tasks.py::test_task_graph_waits_for_all_predecessors_and_skips_suspended_children
multi-predecessor child x once-after-all -> tests/queries/test_tasks.py::test_task_graph_waits_for_all_predecessors_and_skips_suspended_children
manual execution x shared entry point -> tests/queries/test_tasks.py::test_execute_task
## Deviations
Cloud GET_DDL renders allow_overlapping_execution=true for both ALLOW_CHILD_OVERLAP and ALLOW_ALL_OVERLAP. The Cloud snapshots prove that emitted DDL recreates ALLOW_CHILD_OVERLAP for both, so preserving Cloud output precludes a distinct ALLOW_ALL_OVERLAP round-trip.
* LAV-2761: preserve role-safe task execution
Route manual EXECUTE TASK through the pooled PostgreSQL owner while retaining and restoring the caller's session context, so projected roles can use the transaction-controlling task procedure without receiving store write grants. Replace the session overlap fence with a transaction-scoped advisory lock; the committed EXECUTING row remains the durable fence and every error path now releases the lock automatically.
Swept task execution for privileged store access and session-lock leaks; fixed projected-role manual execution and every scheduled overlap-policy acquisition path; clean in ordinary manual execution, scheduled root/child dispatch, and pooled-role restoration.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
## Test matrix
SET/UNSET metadata and errors -> tests/queries/test_tasks.py::test_alter_task_overlap_policy
NO_OVERLAP x active child suppression -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs
ALLOW_CHILD_OVERLAP x active children -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs
ALLOW_CHILD_OVERLAP x active root suppression -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs
ALLOW_ALL_OVERLAP x active roots -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs
graph/run grouping x concurrent states -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs
skipped occurrence x terminal outcomes -> tests/queries/test_tasks.py::test_task_overlap_policy_scheduled_runs
suspended child x scheduled graph -> tests/queries/test_tasks.py::test_task_graph_waits_for_all_predecessors_and_skips_suspended_children
multi-predecessor child x once-after-all -> tests/queries/test_tasks.py::test_task_graph_waits_for_all_predecessors_and_skips_suspended_children
manual execution x account administrator -> tests/queries/test_tasks.py::test_execute_task
manual execution x projected owner role -> tests/queries/test_tasks.py::test_execute_task_under_projected_role
exceptional overlap path x lock cleanup -> covered by transaction-scoped PostgreSQL advisory-lock lifetime
## Deviations
Cloud GET_DDL renders allow_overlapping_execution=true for both ALLOW_CHILD_OVERLAP and ALLOW_ALL_OVERLAP. The Cloud snapshots prove that emitted DDL recreates ALLOW_CHILD_OVERLAP for both, so preserving Cloud output precludes a distinct ALLOW_ALL_OVERLAP round-trip.
---------
Co-authored-by: spiral <spiral@localhost>
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>1 parent 32e9ec1 commit e0daaf3
3 files changed
Lines changed: 22 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15733 | 15733 | | |
15734 | 15734 | | |
15735 | 15735 | | |
| 15736 | + | |
| 15737 | + | |
| 15738 | + | |
| 15739 | + | |
15736 | 15740 | | |
15737 | 15741 | | |
15738 | 15742 | | |
| |||
15761 | 15765 | | |
15762 | 15766 | | |
15763 | 15767 | | |
| 15768 | + | |
| 15769 | + | |
| 15770 | + | |
| 15771 | + | |
15764 | 15772 | | |
15765 | 15773 | | |
15766 | 15774 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
823 | 823 | | |
824 | 824 | | |
825 | 825 | | |
| 826 | + | |
826 | 827 | | |
827 | 828 | | |
828 | 829 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12904 | 12904 | | |
12905 | 12905 | | |
12906 | 12906 | | |
| 12907 | + | |
| 12908 | + | |
| 12909 | + | |
| 12910 | + | |
| 12911 | + | |
| 12912 | + | |
| 12913 | + | |
| 12914 | + | |
| 12915 | + | |
12907 | 12916 | | |
12908 | 12917 | | |
12909 | | - | |
| 12918 | + | |
12910 | 12919 | | |
12911 | 12920 | | |
12912 | 12921 | | |
| |||
12915 | 12924 | | |
12916 | 12925 | | |
12917 | 12926 | | |
| 12927 | + | |
| 12928 | + | |
12918 | 12929 | | |
12919 | 12930 | | |
12920 | | - | |
| 12931 | + | |
12921 | 12932 | | |
12922 | 12933 | | |
12923 | 12934 | | |
| |||
0 commit comments