Commit a73cb6e
LAV-2592: Support CREATE TABLE aggregation policies (#3034)
* LAV-2592: support aggregation policy table lifecycle
Adds Cloud-captured policy lifecycle, table attachment, entity-key parsing, metadata projection, GET_DDL reconstruction, and rename/drop/replace coherence.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
## Deviations
- Query enforcement remains to be completed; the lifecycle and metadata foundation is committed separately.
* LAV-2592: fix vendored sqlparser checks
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: enforce aggregation policies
Implements FORCE conflict protection, complete policy-body and entity-key parsing, SHOW comment metadata, lifecycle metadata coherence, and planner-time aggregation enforcement with entity-key group counting.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
## Deviations
- Masked volatile `created_on` in `tests/queries/ddl/test_governance_policies.py`, following the existing mask in `tests/queries/ddl/test_backup_policies.py:6`; snapshots were re-recorded against the Enterprise Cloud account.
* LAV-2592: wip (budget exhausted: wall_clock)
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: close aggregation policy enforcement gaps
Reject malformed ALTER entity-key lists, enforce ungrouped aggregates without fail-open fallbacks, and match dropped-table POLICY_REFERENCES behavior. Enterprise snapshots cover the added syntax cases.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: enforce aggregate policies across query shapes
Preserve top-level CTEs during aggregation rewriting, qualify each protected relation's entity keys, and require every attached policy boundary in multi-table aggregates. Restore excluded view unsupported-feature coverage and add Enterprise-captured CTE and distinct-key join snapshots.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: preserve aggregate query source boundaries
Keep top-level LIMIT/OFFSET/FETCH clauses outside reconstructed grouping expressions and qualify entity keys only when the base alias is visible at the outer source level. Restore Standard projection-policy rejection coverage and add Enterprise snapshots for LIMIT and nested aliases.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: preserve aggregation policy HAVING clauses
Treat HAVING as a top-level reconstruction boundary, preserve it in the aggregation CTE, and match Cloud suppression of prohibited remainder rows. Replace aggregation-policy error scaffolding with structured run_snapshot_test snapshots captured from the appropriate Cloud accounts.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: preserve aggregate policy ordering
Expose reconstructed projection names and resolve ORDER BY expressions back to their projected ordinals, including transformed aggregate calls. Enterprise snapshots cover projected names, aliases, and aggregate expressions.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: preserve implicit aggregate aliases
Recognize implicit projection aliases during aggregation-policy reconstruction so output names and ORDER BY resolution remain visible. Add an Enterprise-captured regression for implicit grouped and aggregate aliases.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: classify suppressed aggregate projections
Match grouping projections by normalized expression or output alias instead of assuming they lead the select list. Preserve SUM projections in suppressed entity-key remainder rows and cover reordered output with an Enterprise-captured snapshot.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: resolve entity key grouping references
Resolve GROUP BY projection aliases and ordinals before classifying entity-key groups. Enterprise snapshots cover the suppressed remainder total for both forms.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: preserve commas inside quoted expressions
Track SQL quote state, including doubled quote escapes, while splitting aggregate projections and grouping expressions. Enterprise snapshots cover comma-bearing literals and quoted output identifiers.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: preserve dollar-quoted aggregate literals
Treat dollar-quoted values as Snowflake varchar literals, preserve their commas while scanning aggregate expressions, and retain constant grouping projections in suppressed rows. Add an Enterprise-captured regression for a comma-bearing dollar-quoted literal.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: enforce aggregation policies across set queries
Recognize aggregate set-operation query roots, rewrite each top-level arm independently, and preserve compound ordering and set modifiers. Add Enterprise snapshots for UNION, INTERSECT, and EXCEPT.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: scope policies to set operands
Resolve aggregation-policy attachments independently for each set-operation arm, including protected CTE sources, instead of applying every query policy globally. Add Enterprise snapshots for mixed protected/unprotected arms, distinct protected tables, and a shared CTE prefix.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: resolve set policies by relation
Select aggregation policies for each set operand from its referenced table or CTE relation rather than globally reused alias text. Add an Enterprise snapshot proving distinct protected tables may reuse the same alias across UNION operands.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: resolve set policies by relation identity
Retain database and schema identity for aggregation-policy attachments and select each set operand's policy at the qualification required to disambiguate same-named tables. Add an Enterprise snapshot covering distinct entity keys on same-named protected tables across schemas.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: preserve CTE relation identity
Resolve protected CTE sources against the complete attachment identity set and remove the same-name fallback that leaked policies across set operands. Add an Enterprise snapshot for same-named tables in distinct schemas accessed through separate CTEs.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: normalize CTE policy references
Extract quoted and column-list CTE declaration identifiers before matching set-operation operands. Add Enterprise snapshots proving both CTE forms retain aggregation-policy enforcement.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: enforce policies through recursive CTEs
Normalize the optional RECURSIVE modifier before resolving CTE-backed set operands. Add an Enterprise snapshot proving a below-boundary protected arm remains suppressed.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: resolve policies through chained CTEs
Compute transitive CTE dependencies when selecting aggregation policies for each set operand. Add an Enterprise snapshot covering a protected relation behind two CTE layers.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: resolve CTE dependencies from sources
Restrict transitive aggregation-policy propagation to CTE names used as FROM or JOIN relations, preventing projected aliases from creating false dependencies. Add an Enterprise snapshot for a CTE-name/column-alias collision.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: resolve final CTE operands from sources
Restrict final set-operand CTE policy selection to FROM and JOIN relations so projection aliases cannot inherit unrelated policies. Add an Enterprise snapshot covering a protected CTE name reused as an alias in an unprotected arm.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: resolve comma-separated CTE sources
Track source-clause nesting while resolving CTE dependencies so relations following commas inherit aggregation policies. Add an Enterprise snapshot covering a protected CTE after an unprotected comma source in a set operand.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: resolve direct policies from sources
Restrict direct aggregation-policy matching to FROM and JOIN relation identities so projection aliases cannot make an unprotected set arm inherit a protected table's policy. Add an Enterprise snapshot for the alias collision.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: enforce parenthesized set operands
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: enforce wrapped compound queries
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: fix merged compat regressions
Align the anonymous-block helper with dollar-quoted snowflake_varchar arguments and scope the POLICY_REFERENCES missing-table SQLSTATE override so unrelated missing tables retain their Cloud mapping.
Swept the CI failures for the shared missing-table mapping pattern; POLICY_REFERENCES keeps its 02000 result while CREATE OR ALTER STREAM returns 42S02, and anonymous-block callers share the corrected custom-type signature.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
## Test matrix
anonymous block × success -> tests/queries/test_time_travel.py::test_time_travel_execute_immediate_in_window
anonymous block × error -> tests/queries/test_time_travel.py::test_time_travel_anonymous_block_outside_window
missing stream source × error -> tests/queries/test_streams.py::test_create_or_alter_stream_errors_and_unsupported_forms
missing POLICY_REFERENCES table × error -> tests/queries/ddl/test_governance_policies.py::test_aggregation_policy_table_protection
* LAV-2592: preserve governance under projected roles
Route aggregation-policy planner lookups through security-definer helpers so projected share roles never read the private policy store directly. Treat views and dynamic tables as existing TABLE-domain entities for POLICY_REFERENCES, preserving Cloud-compatible view lookup behavior.
Swept planner-hook policy-store reads for the projected-role privilege pattern; both the live-attachment gate and per-table lookup now use privileged internal helpers. Swept TABLE-domain POLICY_REFERENCES entities for table-like catalogs; ordinary tables, views, and dynamic tables are accepted while missing entities retain the Snowflake error.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
## Test matrix
projected role × aggregation-policy gate -> tests/queries/test_samples.py::test_import_and_query_sample_data
protected aggregate × enforcement -> tests/queries/ddl/test_governance_policies.py::test_aggregation_policy_enforcement_shapes
TABLE-domain ordinary table × lookup -> tests/queries/ddl/test_governance_policies.py::test_aggregation_policy_table_lifecycle
TABLE-domain view × lookup -> tests/queries/test_policy_references.py::test_tag_masked_view_reports_as_table
TABLE-domain missing entity × error -> tests/queries/ddl/test_governance_policies.py::test_aggregation_policy_table_protection
* LAV-2592: fix aggregation policy CI regressions
Resolve custom-type dispatch in privileged policy and anonymous-procedure helpers, restore dynamic-table aggregation rejection, and match Cloud error metadata for aggregation lifecycle failures. Remove duplicated post-merge view rejection assertions covered by the same snapshots.
Swept the CI failures for custom snowflake_varchar/native operator mismatches; privileged table-policy lookup and anonymous procedure execution now cast only at native PostgreSQL boundaries. Swept aggregation-policy error shaping for missing policies, invalid keys, duplicate attachments, enforcement, and protected drops.
test_edit_waiver: tests/queries/ddl/test_governance_policies.py
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
## Test matrix
privileged policy lookup × ordinary/shared table -> tests/queries/test_samples.py::test_sample_data_spot_check_customer
anonymous procedure × scalar/table/error paths -> tests/queries/test_anonymous_procedures.py
aggregation validation × missing/duplicate/invalid key -> tests/queries/ddl/test_governance_policies.py::test_aggregation_policy_validation_and_force
aggregation lifecycle × enforcement/protected drop -> tests/queries/ddl/test_governance_policies.py::test_aggregation_policy_table_lifecycle
dynamic table × aggregation policy/entity key rejection -> tests/queries/test_dynamic_table_create_governance.py::test_dynamic_table_create_unsupported_governance
view × aggregation/projection rejection -> tests/queries/ddl/test_governance_policies.py::test_alter_view_unset_aggregation_policy_rejected
view column × projection rejection -> tests/queries/ddl/test_governance_policies.py::test_alter_view_column_unset_projection_policy_rejected
* LAV-2592: investigator: fix recursive CTE policy selection
CI at 7bcb806d8 has one deterministic failure: test_aggregation_policy_same_table_name_across_schemas query-result-5. The WITH RECURSIVE left operand selects one protected entity (expected suppressed NULL/NULL) but returns raw ('a', 10), while the right protected operand remains correctly aggregated as ('b', 70). Trace recursive CTE dependency/policy selection in aggregation_sql/operand_uses_policy around masking.rs:958; do not re-record the authoritative Enterprise snapshot. Rebase or merge the two current origin/main commits first, then run this targeted compat test.
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
* LAV-2592: resolve CTE policies by physical schema
Match aggregation-policy relations against the shared physical schema encoding so protected CTE operands retain their policy after logical names are rewritten for PostgreSQL. This fixes the filtered recursive operand without changing the authoritative Enterprise snapshot.\n\nSwept CTE policy selection for the physical/logical identity mismatch; fixed recursive, direct, quoted, column-list, chained, and same-table-name cross-schema operands through the shared matcher.\n\n## Test matrix\nlogical schema CTE × permitted aggregate -> tests/queries/ddl/test_governance_policies.py::test_aggregation_policy_same_table_name_across_schemas\nphysical schema CTE × minimum boundary -> tests/queries/ddl/test_governance_policies.py::test_aggregation_policy_same_table_name_across_schemas\nrecursive filtered CTE × suppressed aggregate -> tests/queries/ddl/test_governance_policies.py::test_aggregation_policy_same_table_name_across_schemas\nquoted/column-list CTE × protected operand -> tests/queries/ddl/test_governance_policies.py::test_aggregation_policy_same_table_name_across_schemas\nchained CTE × protected operand -> tests/queries/ddl/test_governance_policies.py::test_aggregation_policy_same_table_name_across_schemas\nsame table name across schemas × independent policies -> tests/queries/ddl/test_governance_policies.py::test_aggregation_policy_same_table_name_across_schemas
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
---------
Co-authored-by: spiral <spiral@localhost>
Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>1 parent 6218658 commit a73cb6e
8 files changed
Lines changed: 34 additions & 0 deletions
File tree
- src
- ast
- helpers
- dialect
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4068 | 4068 | | |
4069 | 4069 | | |
4070 | 4070 | | |
| 4071 | + | |
| 4072 | + | |
4071 | 4073 | | |
4072 | 4074 | | |
4073 | 4075 | | |
| |||
4497 | 4499 | | |
4498 | 4500 | | |
4499 | 4501 | | |
| 4502 | + | |
| 4503 | + | |
| 4504 | + | |
4500 | 4505 | | |
4501 | 4506 | | |
4502 | 4507 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
162 | 162 | | |
163 | 163 | | |
164 | 164 | | |
| 165 | + | |
| 166 | + | |
165 | 167 | | |
166 | 168 | | |
167 | 169 | | |
| |||
276 | 278 | | |
277 | 279 | | |
278 | 280 | | |
| 281 | + | |
279 | 282 | | |
280 | 283 | | |
281 | 284 | | |
| |||
550 | 553 | | |
551 | 554 | | |
552 | 555 | | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
553 | 561 | | |
554 | 562 | | |
555 | 563 | | |
| |||
745 | 753 | | |
746 | 754 | | |
747 | 755 | | |
| 756 | + | |
748 | 757 | | |
749 | 758 | | |
750 | 759 | | |
| |||
845 | 854 | | |
846 | 855 | | |
847 | 856 | | |
| 857 | + | |
848 | 858 | | |
849 | 859 | | |
850 | 860 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
751 | 751 | | |
752 | 752 | | |
753 | 753 | | |
| 754 | + | |
754 | 755 | | |
755 | 756 | | |
756 | 757 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2769 | 2769 | | |
2770 | 2770 | | |
2771 | 2771 | | |
| 2772 | + | |
| 2773 | + | |
| 2774 | + | |
| 2775 | + | |
| 2776 | + | |
| 2777 | + | |
| 2778 | + | |
| 2779 | + | |
| 2780 | + | |
| 2781 | + | |
| 2782 | + | |
| 2783 | + | |
| 2784 | + | |
2772 | 2785 | | |
2773 | 2786 | | |
2774 | 2787 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
411 | 411 | | |
412 | 412 | | |
413 | 413 | | |
| 414 | + | |
414 | 415 | | |
415 | 416 | | |
416 | 417 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
781 | 781 | | |
782 | 782 | | |
783 | 783 | | |
| 784 | + | |
784 | 785 | | |
785 | 786 | | |
786 | 787 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2011 | 2011 | | |
2012 | 2012 | | |
2013 | 2013 | | |
| 2014 | + | |
2014 | 2015 | | |
2015 | 2016 | | |
2016 | 2017 | | |
| |||
2204 | 2205 | | |
2205 | 2206 | | |
2206 | 2207 | | |
| 2208 | + | |
2207 | 2209 | | |
2208 | 2210 | | |
2209 | 2211 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6731 | 6731 | | |
6732 | 6732 | | |
6733 | 6733 | | |
| 6734 | + | |
6734 | 6735 | | |
6735 | 6736 | | |
6736 | 6737 | | |
| |||
0 commit comments