Skip to content

Commit a73cb6e

Browse files
localstack-spiral[bot]spiralsabir-akhadov-localstack
authored
LAV-2592: Support CREATE TABLE aggregation policies (#3034)
* LAV-2592: support aggregation policy table lifecycle Adds Cloud-captured policy lifecycle, table attachment, entity-key parsing, metadata projection, GET_DDL reconstruction, and rename/drop/replace coherence. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> ## Deviations - Query enforcement remains to be completed; the lifecycle and metadata foundation is committed separately. * LAV-2592: fix vendored sqlparser checks Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: enforce aggregation policies Implements FORCE conflict protection, complete policy-body and entity-key parsing, SHOW comment metadata, lifecycle metadata coherence, and planner-time aggregation enforcement with entity-key group counting. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> ## Deviations - Masked volatile `created_on` in `tests/queries/ddl/test_governance_policies.py`, following the existing mask in `tests/queries/ddl/test_backup_policies.py:6`; snapshots were re-recorded against the Enterprise Cloud account. * LAV-2592: wip (budget exhausted: wall_clock) Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: close aggregation policy enforcement gaps Reject malformed ALTER entity-key lists, enforce ungrouped aggregates without fail-open fallbacks, and match dropped-table POLICY_REFERENCES behavior. Enterprise snapshots cover the added syntax cases. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: enforce aggregate policies across query shapes Preserve top-level CTEs during aggregation rewriting, qualify each protected relation's entity keys, and require every attached policy boundary in multi-table aggregates. Restore excluded view unsupported-feature coverage and add Enterprise-captured CTE and distinct-key join snapshots. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: preserve aggregate query source boundaries Keep top-level LIMIT/OFFSET/FETCH clauses outside reconstructed grouping expressions and qualify entity keys only when the base alias is visible at the outer source level. Restore Standard projection-policy rejection coverage and add Enterprise snapshots for LIMIT and nested aliases. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: preserve aggregation policy HAVING clauses Treat HAVING as a top-level reconstruction boundary, preserve it in the aggregation CTE, and match Cloud suppression of prohibited remainder rows. Replace aggregation-policy error scaffolding with structured run_snapshot_test snapshots captured from the appropriate Cloud accounts. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: preserve aggregate policy ordering Expose reconstructed projection names and resolve ORDER BY expressions back to their projected ordinals, including transformed aggregate calls. Enterprise snapshots cover projected names, aliases, and aggregate expressions. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: preserve implicit aggregate aliases Recognize implicit projection aliases during aggregation-policy reconstruction so output names and ORDER BY resolution remain visible. Add an Enterprise-captured regression for implicit grouped and aggregate aliases. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: classify suppressed aggregate projections Match grouping projections by normalized expression or output alias instead of assuming they lead the select list. Preserve SUM projections in suppressed entity-key remainder rows and cover reordered output with an Enterprise-captured snapshot. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: resolve entity key grouping references Resolve GROUP BY projection aliases and ordinals before classifying entity-key groups. Enterprise snapshots cover the suppressed remainder total for both forms. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: preserve commas inside quoted expressions Track SQL quote state, including doubled quote escapes, while splitting aggregate projections and grouping expressions. Enterprise snapshots cover comma-bearing literals and quoted output identifiers. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: preserve dollar-quoted aggregate literals Treat dollar-quoted values as Snowflake varchar literals, preserve their commas while scanning aggregate expressions, and retain constant grouping projections in suppressed rows. Add an Enterprise-captured regression for a comma-bearing dollar-quoted literal. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: enforce aggregation policies across set queries Recognize aggregate set-operation query roots, rewrite each top-level arm independently, and preserve compound ordering and set modifiers. Add Enterprise snapshots for UNION, INTERSECT, and EXCEPT. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: scope policies to set operands Resolve aggregation-policy attachments independently for each set-operation arm, including protected CTE sources, instead of applying every query policy globally. Add Enterprise snapshots for mixed protected/unprotected arms, distinct protected tables, and a shared CTE prefix. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: resolve set policies by relation Select aggregation policies for each set operand from its referenced table or CTE relation rather than globally reused alias text. Add an Enterprise snapshot proving distinct protected tables may reuse the same alias across UNION operands. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: resolve set policies by relation identity Retain database and schema identity for aggregation-policy attachments and select each set operand's policy at the qualification required to disambiguate same-named tables. Add an Enterprise snapshot covering distinct entity keys on same-named protected tables across schemas. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: preserve CTE relation identity Resolve protected CTE sources against the complete attachment identity set and remove the same-name fallback that leaked policies across set operands. Add an Enterprise snapshot for same-named tables in distinct schemas accessed through separate CTEs. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: normalize CTE policy references Extract quoted and column-list CTE declaration identifiers before matching set-operation operands. Add Enterprise snapshots proving both CTE forms retain aggregation-policy enforcement. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: enforce policies through recursive CTEs Normalize the optional RECURSIVE modifier before resolving CTE-backed set operands. Add an Enterprise snapshot proving a below-boundary protected arm remains suppressed. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: resolve policies through chained CTEs Compute transitive CTE dependencies when selecting aggregation policies for each set operand. Add an Enterprise snapshot covering a protected relation behind two CTE layers. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: resolve CTE dependencies from sources Restrict transitive aggregation-policy propagation to CTE names used as FROM or JOIN relations, preventing projected aliases from creating false dependencies. Add an Enterprise snapshot for a CTE-name/column-alias collision. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: resolve final CTE operands from sources Restrict final set-operand CTE policy selection to FROM and JOIN relations so projection aliases cannot inherit unrelated policies. Add an Enterprise snapshot covering a protected CTE name reused as an alias in an unprotected arm. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: resolve comma-separated CTE sources Track source-clause nesting while resolving CTE dependencies so relations following commas inherit aggregation policies. Add an Enterprise snapshot covering a protected CTE after an unprotected comma source in a set operand. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: resolve direct policies from sources Restrict direct aggregation-policy matching to FROM and JOIN relation identities so projection aliases cannot make an unprotected set arm inherit a protected table's policy. Add an Enterprise snapshot for the alias collision. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: enforce parenthesized set operands Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: enforce wrapped compound queries Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: fix merged compat regressions Align the anonymous-block helper with dollar-quoted snowflake_varchar arguments and scope the POLICY_REFERENCES missing-table SQLSTATE override so unrelated missing tables retain their Cloud mapping. Swept the CI failures for the shared missing-table mapping pattern; POLICY_REFERENCES keeps its 02000 result while CREATE OR ALTER STREAM returns 42S02, and anonymous-block callers share the corrected custom-type signature. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> ## Test matrix anonymous block × success -> tests/queries/test_time_travel.py::test_time_travel_execute_immediate_in_window anonymous block × error -> tests/queries/test_time_travel.py::test_time_travel_anonymous_block_outside_window missing stream source × error -> tests/queries/test_streams.py::test_create_or_alter_stream_errors_and_unsupported_forms missing POLICY_REFERENCES table × error -> tests/queries/ddl/test_governance_policies.py::test_aggregation_policy_table_protection * LAV-2592: preserve governance under projected roles Route aggregation-policy planner lookups through security-definer helpers so projected share roles never read the private policy store directly. Treat views and dynamic tables as existing TABLE-domain entities for POLICY_REFERENCES, preserving Cloud-compatible view lookup behavior. Swept planner-hook policy-store reads for the projected-role privilege pattern; both the live-attachment gate and per-table lookup now use privileged internal helpers. Swept TABLE-domain POLICY_REFERENCES entities for table-like catalogs; ordinary tables, views, and dynamic tables are accepted while missing entities retain the Snowflake error. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> ## Test matrix projected role × aggregation-policy gate -> tests/queries/test_samples.py::test_import_and_query_sample_data protected aggregate × enforcement -> tests/queries/ddl/test_governance_policies.py::test_aggregation_policy_enforcement_shapes TABLE-domain ordinary table × lookup -> tests/queries/ddl/test_governance_policies.py::test_aggregation_policy_table_lifecycle TABLE-domain view × lookup -> tests/queries/test_policy_references.py::test_tag_masked_view_reports_as_table TABLE-domain missing entity × error -> tests/queries/ddl/test_governance_policies.py::test_aggregation_policy_table_protection * LAV-2592: fix aggregation policy CI regressions Resolve custom-type dispatch in privileged policy and anonymous-procedure helpers, restore dynamic-table aggregation rejection, and match Cloud error metadata for aggregation lifecycle failures. Remove duplicated post-merge view rejection assertions covered by the same snapshots. Swept the CI failures for custom snowflake_varchar/native operator mismatches; privileged table-policy lookup and anonymous procedure execution now cast only at native PostgreSQL boundaries. Swept aggregation-policy error shaping for missing policies, invalid keys, duplicate attachments, enforcement, and protected drops. test_edit_waiver: tests/queries/ddl/test_governance_policies.py Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> ## Test matrix privileged policy lookup × ordinary/shared table -> tests/queries/test_samples.py::test_sample_data_spot_check_customer anonymous procedure × scalar/table/error paths -> tests/queries/test_anonymous_procedures.py aggregation validation × missing/duplicate/invalid key -> tests/queries/ddl/test_governance_policies.py::test_aggregation_policy_validation_and_force aggregation lifecycle × enforcement/protected drop -> tests/queries/ddl/test_governance_policies.py::test_aggregation_policy_table_lifecycle dynamic table × aggregation policy/entity key rejection -> tests/queries/test_dynamic_table_create_governance.py::test_dynamic_table_create_unsupported_governance view × aggregation/projection rejection -> tests/queries/ddl/test_governance_policies.py::test_alter_view_unset_aggregation_policy_rejected view column × projection rejection -> tests/queries/ddl/test_governance_policies.py::test_alter_view_column_unset_projection_policy_rejected * LAV-2592: investigator: fix recursive CTE policy selection CI at 7bcb806d8 has one deterministic failure: test_aggregation_policy_same_table_name_across_schemas query-result-5. The WITH RECURSIVE left operand selects one protected entity (expected suppressed NULL/NULL) but returns raw ('a', 10), while the right protected operand remains correctly aggregated as ('b', 70). Trace recursive CTE dependency/policy selection in aggregation_sql/operand_uses_policy around masking.rs:958; do not re-record the authoritative Enterprise snapshot. Rebase or merge the two current origin/main commits first, then run this targeted compat test. Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> * LAV-2592: resolve CTE policies by physical schema Match aggregation-policy relations against the shared physical schema encoding so protected CTE operands retain their policy after logical names are rewritten for PostgreSQL. This fixes the filtered recursive operand without changing the authoritative Enterprise snapshot.\n\nSwept CTE policy selection for the physical/logical identity mismatch; fixed recursive, direct, quoted, column-list, chained, and same-table-name cross-schema operands through the shared matcher.\n\n## Test matrix\nlogical schema CTE × permitted aggregate -> tests/queries/ddl/test_governance_policies.py::test_aggregation_policy_same_table_name_across_schemas\nphysical schema CTE × minimum boundary -> tests/queries/ddl/test_governance_policies.py::test_aggregation_policy_same_table_name_across_schemas\nrecursive filtered CTE × suppressed aggregate -> tests/queries/ddl/test_governance_policies.py::test_aggregation_policy_same_table_name_across_schemas\nquoted/column-list CTE × protected operand -> tests/queries/ddl/test_governance_policies.py::test_aggregation_policy_same_table_name_across_schemas\nchained CTE × protected operand -> tests/queries/ddl/test_governance_policies.py::test_aggregation_policy_same_table_name_across_schemas\nsame table name across schemas × independent policies -> tests/queries/ddl/test_governance_policies.py::test_aggregation_policy_same_table_name_across_schemas Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud> --------- Co-authored-by: spiral <spiral@localhost> Co-authored-by: Sabir Akhadov <sabir.akhadov@localstack.cloud>
1 parent 6218658 commit a73cb6e

8 files changed

Lines changed: 34 additions & 0 deletions

File tree

‎src/ast/ddl.rs‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4068,6 +4068,8 @@ pub struct CreateTable {
40684068
/// Snowflake "WITH AGGREGATION POLICY" clause
40694069
/// <https://docs.snowflake.com/en/sql-reference/sql/create-table>
40704070
pub with_aggregation_policy: Option<ObjectName>,
4071+
/// Snowflake aggregation-policy entity key.
4072+
pub aggregation_policy_entity_key: Vec<Ident>,
40714073
/// Snowflake "WITH ROW ACCESS POLICY" clause
40724074
/// <https://docs.snowflake.com/en/sql-reference/sql/create-table>
40734075
pub with_row_access_policy: Option<RowAccessPolicy>,
@@ -4497,6 +4499,9 @@ impl fmt::Display for CreateTable {
44974499

44984500
if let Some(with_aggregation_policy) = &self.with_aggregation_policy {
44994501
write!(f, " WITH AGGREGATION POLICY {with_aggregation_policy}",)?;
4502+
if !self.aggregation_policy_entity_key.is_empty() {
4503+
write!(f, " ENTITY KEY ({})", display_comma_separated(&self.aggregation_policy_entity_key))?;
4504+
}
45004505
}
45014506

45024507
if let Some(row_access_policy) = &self.with_row_access_policy {

‎src/ast/helpers/stmt_create_table.rs‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -162,6 +162,8 @@ pub struct CreateTableBuilder {
162162
pub iceberg_default_ddl_collation: Option<String>,
163163
/// Optional aggregation policy object name.
164164
pub with_aggregation_policy: Option<ObjectName>,
165+
/// Optional entity-key columns for the aggregation policy.
166+
pub aggregation_policy_entity_key: Vec<Ident>,
165167
/// Optional row access policy applied to the table.
166168
pub with_row_access_policy: Option<RowAccessPolicy>,
167169
/// Optional storage lifecycle policy applied to the table.
@@ -276,6 +278,7 @@ impl CreateTableBuilder {
276278
default_ddl_collation: None,
277279
iceberg_default_ddl_collation: None,
278280
with_aggregation_policy: None,
281+
aggregation_policy_entity_key: vec![],
279282
with_row_access_policy: None,
280283
with_storage_lifecycle_policy: None,
281284
with_tags: None,
@@ -550,6 +553,11 @@ impl CreateTableBuilder {
550553
self.with_aggregation_policy = with_aggregation_policy;
551554
self
552555
}
556+
/// Set aggregation-policy entity-key columns.
557+
pub fn aggregation_policy_entity_key(mut self, columns: Vec<Ident>) -> Self {
558+
self.aggregation_policy_entity_key = columns;
559+
self
560+
}
553561
/// Attach a row access policy to the table.
554562
pub fn with_row_access_policy(
555563
mut self,
@@ -745,6 +753,7 @@ impl CreateTableBuilder {
745753
default_ddl_collation: self.default_ddl_collation,
746754
iceberg_default_ddl_collation: self.iceberg_default_ddl_collation,
747755
with_aggregation_policy: self.with_aggregation_policy,
756+
aggregation_policy_entity_key: self.aggregation_policy_entity_key,
748757
with_row_access_policy: self.with_row_access_policy,
749758
with_storage_lifecycle_policy: self.with_storage_lifecycle_policy,
750759
with_tags: self.with_tags,
@@ -845,6 +854,7 @@ impl From<CreateTable> for CreateTableBuilder {
845854
default_ddl_collation: table.default_ddl_collation,
846855
iceberg_default_ddl_collation: table.iceberg_default_ddl_collation,
847856
with_aggregation_policy: table.with_aggregation_policy,
857+
aggregation_policy_entity_key: table.aggregation_policy_entity_key,
848858
with_row_access_policy: table.with_row_access_policy,
849859
with_storage_lifecycle_policy: table.with_storage_lifecycle_policy,
850860
with_tags: table.with_tags,

‎src/ast/spans.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -751,6 +751,7 @@ impl Spanned for CreateTable {
751751
default_ddl_collation: _, // string, no span
752752
iceberg_default_ddl_collation: _, // string, no span
753753
with_aggregation_policy: _, // todo, Snowflake specific
754+
aggregation_policy_entity_key: _, // todo, Snowflake specific
754755
with_row_access_policy: _, // todo, Snowflake specific
755756
with_storage_lifecycle_policy: _, // todo, Snowflake specific
756757
with_tags: _, // todo, Snowflake specific

‎src/dialect/snowflake.rs‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2769,6 +2769,19 @@ pub fn parse_create_table(
27692769
parser.expect_keyword_is(Keyword::POLICY)?;
27702770
let aggregation_policy = parser.parse_object_name(false)?;
27712771
builder = builder.with_aggregation_policy(Some(aggregation_policy));
2772+
if parser.parse_keywords(&[Keyword::ENTITY, Keyword::KEY]) {
2773+
parser.expect_token(&Token::LParen)?;
2774+
let columns = parser.parse_comma_separated(|p| p.parse_identifier())?;
2775+
parser.expect_token(&Token::RParen)?;
2776+
builder = builder.aggregation_policy_entity_key(columns);
2777+
}
2778+
}
2779+
Keyword::ENTITY if builder.with_aggregation_policy.is_some() => {
2780+
parser.expect_keyword_is(Keyword::KEY)?;
2781+
parser.expect_token(&Token::LParen)?;
2782+
let columns = parser.parse_comma_separated(|p| p.parse_identifier())?;
2783+
parser.expect_token(&Token::RParen)?;
2784+
builder = builder.aggregation_policy_entity_key(columns);
27722785
}
27732786
Keyword::ROW => {
27742787
parser.expect_keywords(&[Keyword::ACCESS, Keyword::POLICY])?;

‎src/keywords.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -411,6 +411,7 @@ define_keywords!(
411411
ENGINE,
412412
ENGINE_ATTRIBUTE,
413413
ENROLL,
414+
ENTITY,
414415
ENUM,
415416
ENUM16,
416417
ENUM8,

‎tests/sqlparser_duckdb.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -781,6 +781,7 @@ fn test_duckdb_union_datatype() {
781781
default_ddl_collation: Default::default(),
782782
iceberg_default_ddl_collation: Default::default(),
783783
with_aggregation_policy: Default::default(),
784+
aggregation_policy_entity_key: Default::default(),
784785
with_row_access_policy: Default::default(),
785786
with_storage_lifecycle_policy: Default::default(),
786787
with_tags: Default::default(),

‎tests/sqlparser_mssql.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2011,6 +2011,7 @@ fn parse_create_table_with_valid_options() {
20112011
default_ddl_collation: None,
20122012
iceberg_default_ddl_collation: None,
20132013
with_aggregation_policy: None,
2014+
aggregation_policy_entity_key: vec![],
20142015
with_row_access_policy: None,
20152016
with_storage_lifecycle_policy: None,
20162017
with_tags: None,
@@ -2204,6 +2205,7 @@ fn parse_create_table_with_identity_column() {
22042205
default_ddl_collation: None,
22052206
iceberg_default_ddl_collation: None,
22062207
with_aggregation_policy: None,
2208+
aggregation_policy_entity_key: vec![],
22072209
with_row_access_policy: None,
22082210
with_storage_lifecycle_policy: None,
22092211
with_tags: None,

‎tests/sqlparser_postgres.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6731,6 +6731,7 @@ fn parse_trigger_related_functions() {
67316731
default_ddl_collation: None,
67326732
iceberg_default_ddl_collation: None,
67336733
with_aggregation_policy: None,
6734+
aggregation_policy_entity_key: vec![],
67346735
with_row_access_policy: None,
67356736
with_storage_lifecycle_policy: None,
67366737
with_tags: None,

0 commit comments

Comments
 (0)