From 8e88d3b249e07e662f410525c1c064b5ff8b77d8 Mon Sep 17 00:00:00 2001 From: Huaxin Gao Date: Wed, 26 Aug 2026 12:52:00 +0800 Subject: [PATCH 01/25] Spec: Add table constraints --- format/spec.md | 126 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 126 insertions(+) diff --git a/format/spec.md b/format/spec.md index 3492de72a8b0..fc172347171a 100644 --- a/format/spec.md +++ b/format/spec.md @@ -62,6 +62,7 @@ The full set of changes are listed in [Appendix E](#version-3). Version 4 of the Iceberg spec restructures metadata for improved performance and new capabilities: * Support for [relative locations](#file-locations-in-metadata) in metadata fields +* Support for table [constraints](#constraints) The full set of changes are listed in [Appendix E](#version-4). @@ -654,6 +655,106 @@ Sorting floating-point numbers should produce the following behavior: `-NaN` < ` A data or delete file is associated with a sort order by the sort order's id within [a manifest](#manifests). Therefore, the table must declare all the sort orders for lookup. A table could also be configured with a default sort order id, indicating how the new data should be sorted by default. Writers should use this default sort order to sort the data on write, but are not required to if the default order is prohibitively expensive, as it would be for streaming writes. +### Constraints + +A **constraint** declares a property that a table's rows are expected to satisfy. A constraint's definition is stored in table metadata. Whether a constraint holds is recorded for each snapshot, see [Constraint Validation](#constraint-validation). + +Iceberg does not evaluate constraints. Enforcement and validation are performed by engines that write to a table. Iceberg stores constraint definitions and records the status that a writer reports for a commit without verifying it. + +Constraints are added in v4 and are not supported in v3 or earlier. + +Three constraint types are defined: + +* `check` -- every row must satisfy a predicate +* `unique` -- the values of a set of fields must be distinct across all rows +* `primary-key` -- the values of a set of fields must be distinct across all rows and must not be null + +Constraints are stored separately from schemas because a constraint may reference more than one field and because constraints and schemas evolve independently. Every constraint references the fields that it applies to by field ID, so a constraint continues to apply to the same columns after a column is renamed or reordered. + +A required field in a schema expresses `NOT NULL`. It is not represented as a constraint. + +#### Constraint Fields + +A constraint consists of the following fields: + +| Requirement | Field name | Type | Description | +|-------------|---------------------------|-----------|-------------| +| _required_ | **`constraint-id`** | `int` | ID of the constraint; unique within the table | +| _required_ | **`type`** | `string` | The constraint type: `check`, `unique`, or `primary-key` | +| _required_ | **`name`** | `string` | A name for the constraint that is unique within the table. Names are for human consumption and must not be used to identify a constraint in metadata | +| _required_ | **`enforced`** | `boolean` | Whether writers must verify that the rows they add satisfy the constraint | +| _required_ | **`timestamp-ms`** | `long` | Timestamp in milliseconds from the unix epoch when the constraint was created or last modified | +| _required_ | **`constraint-metadata`** | `struct` | Defines what the constraint requires; its fields depend on the constraint `type` (see below) | + +`constraint-metadata` for a `check` constraint consists of: + +| Requirement | Field name | Type | Description | +|-------------|------------------|--------------|-------------| +| _required_ | **`expression`** | `expression` | A predicate that every row must satisfy, see [Check Constraint Expressions](#check-constraint-expressions) | + +`constraint-metadata` for a `unique` or `primary-key` constraint consists of: + +| Requirement | Field name | Type | Description | +|-------------|-----------------|-------------|-------------| +| _required_ | **`field-ids`** | `list` | A list of field IDs that the constraint applies to | + +Each type carries only the metadata that it requires: a `check` constraint must not declare `field-ids` and a `unique` or `primary-key` constraint must not declare an `expression`. This keeps a single source of truth for the fields that a constraint references. + +The `field-ids` of a `unique` or `primary-key` constraint must reference primitive fields that are either top-level fields or nested in required structs, and must not reference fields within a `list` or a `map`. These are the same restrictions that apply to [identifier fields](#identifier-field-ids). + +When a constraint is `enforced`, writers must verify that the rows they add satisfy the constraint and must fail the write if they do not. A writer that cannot verify an enforced constraint must reject writes to the table rather than add rows that have not been verified. When a constraint is not enforced, writers are not required to verify the rows they add. + +Whether to trust a constraint that is not enforced is left to engines and is not tracked in table metadata. + +Constraint IDs are assigned from the table's `last-constraint-id`, which is treated as 0 when it is not present. Writers must assign a new constraint an ID that is higher than the table's current `last-constraint-id` and must update `last-constraint-id` to the highest assigned ID. Constraint IDs must not be reused after the constraint that used an ID is removed, because retained snapshots may still reference the removed ID. Readers must not assume that every `constraint-id` referenced by a snapshot is present in `constraints`. + +#### Check Constraint Expressions + +The `expression` of a `check` constraint is serialized as described in the [Iceberg expressions spec](expressions-spec.md) and must use ID references so that it remains bound to the same fields when columns are renamed or reordered. + +A check expression must be a deterministic predicate over a single row. Expressions that depend on more than one row, such as aggregates and window functions, and expressions that depend on another table, such as subqueries, must not be used. + +Iceberg predicates use two-valued logic: a predicate always produces true or false and never produces null, so a comparison with a null operand produces false. This differs from SQL `CHECK`, where a row satisfies a constraint unless the predicate produces false and a null value therefore satisfies the constraint. + +To express SQL `CHECK` semantics for an optional field, the stored expression must make the null case explicit. For example, SQL `CHECK (price >= 0)` for an optional `price` field is stored as the expression for `price >= 0 OR price IS NULL`. This is unnecessary for required fields, which can never be null. + +#### Constraint Validation + +Enforcement and validation are separate properties. Whether writers must verify the rows that they add is a property of a constraint, tracked by `enforced`. Whether a table is known to satisfy a constraint is a property of a table's data, tracked per snapshot by `constraint-statuses`. + +A snapshot's `constraint-statuses` records the status of each constraint that was active when the snapshot was created. Each entry consists of: + +| Requirement | Field name | Type | Description | +|-------------|---------------------|----------|-------------| +| _required_ | **`constraint-id`** | `int` | ID of the constraint that the status describes | +| _required_ | **`status`** | `string` | The status of the constraint for this snapshot | + +A status is one of: + +| Status | Description | +|---------------|-------------| +| `validated` | The constraint was checked and holds for all rows in the snapshot | +| `valid` | The constraint holds for all rows in the snapshot because it was enforced for the commit that produced the snapshot and the parent snapshot's status is `validated` or `valid` | +| `invalid` | The constraint was checked and at least one row in the snapshot violates it | +| `unvalidated` | Whether the constraint holds for all rows in the snapshot is not known | + +A constraint that is not present in a snapshot's `constraint-statuses` was not active when the snapshot was created, and the snapshot makes no claim about it. This is the case for snapshots created before the constraint was added and for snapshots created after it was dropped. + +Writers must record a status for every constraint that is active when a snapshot is created: + +* `validated` must not be recorded unless the constraint was checked for every row in the snapshot +* `valid` must not be recorded unless the constraint was enforced for the commit and the parent snapshot's status for the constraint is `validated` or `valid` +* `invalid` must not be recorded unless a row in the snapshot is known to violate the constraint +* `unvalidated` must be recorded when no other status can be recorded + +Enforcing a constraint for a commit is not sufficient to record `valid`. When the parent snapshot's status is not `validated` or `valid`, rows added by earlier commits were never checked, so the status is `unvalidated` even though the writer verified the rows that it added. + +When a constraint becomes enforced, either by being added with `enforced` set to true or by `enforced` changing from false to true, writers should validate the table and record `validated`. A writer that does not validate records `unvalidated`, and the constraint remains `unvalidated` until a later validation records `validated`. + +Only `validated` and `valid` state that a constraint holds. They are distinguished so that readers can tell whether that conclusion was reached by checking data or by relying on writers to enforce the constraint, and so that the most recent `validated` snapshot can be found if a constraint is later found to be `invalid`. + +Writers may commit to a table where a constraint is `invalid`. An enforced constraint requires that a writer not add rows that violate the constraint; it does not require a writer to repair existing violations. Returning a constraint to `validated` is a separate operation. + ### Manifests A manifest is an immutable Avro file that lists data files or delete files, along with each file’s partition data tuple, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. Manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot. @@ -962,6 +1063,20 @@ A snapshot consists of the following fields: | | | _required_ | **`first-row-id`** | The first `_row_id` assigned to the first row in the first data file in the first manifest, see [Row Lineage](#row-lineage) | | | | _required_ | **`added-rows`** | The upper bound of the number of rows with assigned row IDs, see [Row Lineage](#row-lineage) | | | | _optional_ | **`key-id`** | ID of the encryption key that encrypts the manifest list key metadata | +=== "v4" + | v4 | Field | Description | + |------------|------------------------------|-------------| + | _required_ | **`snapshot-id`** | A unique long ID | + | _optional_ | **`parent-snapshot-id`** | The snapshot ID of the snapshot's parent. Omitted for any snapshot with no parent | + | _required_ | **`sequence-number`** | A monotonically increasing long that tracks the order of changes to a table | + | _required_ | **`timestamp-ms`** | A timestamp when the snapshot was created, used for garbage collection and table inspection | + | _required_ | **`manifest-list`** | The location of a manifest list for this snapshot that tracks manifest files with additional metadata | + | _required_ | **`summary`** | A string map that summarizes the snapshot changes, including `operation` as a _required_ field (see below) | + | _optional_ | **`schema-id`** | ID of the table's current schema when the snapshot was created | + | _required_ | **`first-row-id`** | The first `_row_id` assigned to the first row in the first data file in the first manifest, see [Row Lineage](#row-lineage) | + | _required_ | **`added-rows`** | The upper bound of the number of rows with assigned row IDs, see [Row Lineage](#row-lineage) | + | _optional_ | **`key-id`** | ID of the encryption key that encrypts the manifest list key metadata | + | _optional_ | **`constraint-statuses`** | A list (optional) of constraint statuses for the snapshot, see [Constraint Validation](#constraint-validation) | The snapshot summary's `operation` field is used by some operations, like snapshot expiration, to skip processing certain snapshots. Possible `operation` values are: @@ -1194,6 +1309,8 @@ Table metadata consists of the following fields: | _optional_ | **`partition-statistics`** | A list (optional) of [partition statistics](#partition-statistics). | | _required_ | **`next-row-id`** | A `long` higher than all assigned row IDs; the next snapshot's `first-row-id`. See [Row Lineage](#row-lineage). | | _optional_ | **`encryption-keys`** | A list (optional) of [encryption keys](#encryption-keys) used for table encryption. | + | _optional_ | **`last-constraint-id`** | An integer; the highest assigned constraint ID for the table. This is used to ensure constraints are always assigned an unused ID. See [Constraints](#constraints). | + | _optional_ | **`constraints`** | A list (optional) of [constraints](#constraints) for the table. | For serialization details, see Appendix C. @@ -1796,6 +1913,8 @@ A metadata JSON file may be compressed with [GZIP](https://datatracker.ietf.org/ |**`default-sort-order-id`**|`JSON int`|`0`| |**`refs`**|`JSON map with string key and object value:`
`{`
  `"": {`
  `"snapshot-id": ,`
  `"type": ,`
  `"max-ref-age-ms": ,`
  `...`
  `}`
  `...`
`}`|`{`
  `"test": {`
  `"snapshot-id": 123456789000,`
  `"type": "tag",`
  `"max-ref-age-ms": 10000000`
  `}`
`}`| |**`encryption-keys`**|`JSON list of encryption key objects`|`[ {"key-id": "5f819b", "key-metadata": "aWNlYmVyZwo="} ]`| +|**`last-constraint-id`**|`JSON int`|`2`| +|**`constraints`**|`JSON list of objects: [ {`
  `"constraint-id": ,`
  `"type": ,`
  `"name": "",`
  `"enforced": ,`
  `"timestamp-ms": ,`
  `"constraint-metadata": { ... }`
  `},`
  `...`
`]`|`[ {`
  `"constraint-id": 1,`
  `"type": "primary-key",`
  `"name": "pk_sales",`
  `"enforced": false,`
  `"timestamp-ms": 1515100955770,`
  `"constraint-metadata": {`
    `"field-ids": [1]`
  `}`
`} ]`| ### Name Mapping Serialization @@ -1913,6 +2032,13 @@ Reading v4 metadata: * Relative paths must be resolved against the table location before use (see [Path Resolution](#path-resolution)) * When `location` is omitted, the table location must be provided (see [Table Location Specification](#table-location-specification)) +Constraints are added in v4: + +* [Constraints](#constraints) must not be added to v3 or earlier tables +* Table metadata may contain `constraints` and `last-constraint-id` +* Snapshots may contain `constraint-statuses` +* Upgrading a v2 or v3 table to v4 does not add constraints; a table upgraded to v4 has no constraints until they are added + ### Version 3 Default values are added to struct fields in v3. From 879534d6381ca3d4407a416647a055748c76cf26 Mon Sep 17 00:00:00 2001 From: Huaxin Gao Date: Wed, 26 Aug 2026 14:47:44 +0800 Subject: [PATCH 02/25] nit --- format/spec.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/format/spec.md b/format/spec.md index fc172347171a..fff4ff9045fb 100644 --- a/format/spec.md +++ b/format/spec.md @@ -669,7 +669,7 @@ Three constraint types are defined: * `unique` -- the values of a set of fields must be distinct across all rows * `primary-key` -- the values of a set of fields must be distinct across all rows and must not be null -Constraints are stored separately from schemas because a constraint may reference more than one field and because constraints and schemas evolve independently. Every constraint references the fields that it applies to by field ID, so a constraint continues to apply to the same columns after a column is renamed or reordered. +Constraints are stored separately from schemas because the two evolve independently. Every constraint references the fields that it applies to by field ID, so a constraint continues to apply to the same columns after a column is renamed or reordered. A required field in a schema expresses `NOT NULL`. It is not represented as a constraint. @@ -753,7 +753,7 @@ When a constraint becomes enforced, either by being added with `enforced` set to Only `validated` and `valid` state that a constraint holds. They are distinguished so that readers can tell whether that conclusion was reached by checking data or by relying on writers to enforce the constraint, and so that the most recent `validated` snapshot can be found if a constraint is later found to be `invalid`. -Writers may commit to a table where a constraint is `invalid`. An enforced constraint requires that a writer not add rows that violate the constraint; it does not require a writer to repair existing violations. Returning a constraint to `validated` is a separate operation. +Writers may commit to a table where a constraint is `invalid`. An enforced constraint requires that a writer not add rows that violate the constraint; it does not require a writer to repair existing violations. ### Manifests From 53d4ae63fbbc341ea4b3758ccdd98d8a2bb3a8f8 Mon Sep 17 00:00:00 2001 From: Huaxin Gao Date: Thu, 27 Aug 2026 11:50:21 +0800 Subject: [PATCH 03/25] more changes --- format/spec.md | 59 +++++++++++++++++++++++++++++++++++--------------- 1 file changed, 41 insertions(+), 18 deletions(-) diff --git a/format/spec.md b/format/spec.md index fff4ff9045fb..394a4001c3bd 100644 --- a/format/spec.md +++ b/format/spec.md @@ -666,7 +666,7 @@ Constraints are added in v4 and are not supported in v3 or earlier. Three constraint types are defined: * `check` -- every row must satisfy a predicate -* `unique` -- the values of a set of fields must be distinct across all rows +* `unique` -- the values of a set of fields must be distinct across all rows; a null value is not equal to any other null value, so more than one row may be null * `primary-key` -- the values of a set of fields must be distinct across all rows and must not be null Constraints are stored separately from schemas because the two evolve independently. Every constraint references the fields that it applies to by field ID, so a constraint continues to apply to the same columns after a column is renamed or reordered. @@ -683,7 +683,7 @@ A constraint consists of the following fields: | _required_ | **`type`** | `string` | The constraint type: `check`, `unique`, or `primary-key` | | _required_ | **`name`** | `string` | A name for the constraint that is unique within the table. Names are for human consumption and must not be used to identify a constraint in metadata | | _required_ | **`enforced`** | `boolean` | Whether writers must verify that the rows they add satisfy the constraint | -| _required_ | **`timestamp-ms`** | `long` | Timestamp in milliseconds from the unix epoch when the constraint was created or last modified | +| _required_ | **`timestamp-ms`** | `long` | Timestamp in milliseconds from the unix epoch when the constraint was created or last modified. The timestamp is informational and must not be used to determine whether a constraint applies to a snapshot or whether it holds | | _required_ | **`constraint-metadata`** | `struct` | Defines what the constraint requires; its fields depend on the constraint `type` (see below) | `constraint-metadata` for a `check` constraint consists of: @@ -722,14 +722,7 @@ To express SQL `CHECK` semantics for an optional field, the stored expression mu Enforcement and validation are separate properties. Whether writers must verify the rows that they add is a property of a constraint, tracked by `enforced`. Whether a table is known to satisfy a constraint is a property of a table's data, tracked per snapshot by `constraint-statuses`. -A snapshot's `constraint-statuses` records the status of each constraint that was active when the snapshot was created. Each entry consists of: - -| Requirement | Field name | Type | Description | -|-------------|---------------------|----------|-------------| -| _required_ | **`constraint-id`** | `int` | ID of the constraint that the status describes | -| _required_ | **`status`** | `string` | The status of the constraint for this snapshot | - -A status is one of: +The status of a constraint for a snapshot is one of: | Status | Description | |---------------|-------------| @@ -738,19 +731,49 @@ A status is one of: | `invalid` | The constraint was checked and at least one row in the snapshot violates it | | `unvalidated` | Whether the constraint holds for all rows in the snapshot is not known | -A constraint that is not present in a snapshot's `constraint-statuses` was not active when the snapshot was created, and the snapshot makes no claim about it. This is the case for snapshots created before the constraint was added and for snapshots created after it was dropped. +A snapshot's `constraint-statuses` consists of: + +| Requirement | Field name | Type | Description | +|-------------|--------------------------|----------------|-------------| +| _required_ | **`last-constraint-id`** | `int` | The table's `last-constraint-id` when the snapshot was created | +| _required_ | **`default-status`** | `string` | The status of every constraint that is not listed in `statuses` | +| _optional_ | **`statuses`** | `list` | Statuses for constraints whose status is not `default-status` | + +Each entry of `statuses` consists of: + +| Requirement | Field name | Type | Description | +|-------------|---------------------|----------|-------------| +| _required_ | **`constraint-id`** | `int` | ID of the constraint that the status describes | +| _required_ | **`status`** | `string` | The status of the constraint for this snapshot | + +`default-status` covers the constraints that share a status and `statuses` records only the constraints that differ from it, so the size of `constraint-statuses` does not grow with the number of constraints in a table. When every constraint has the same status, that status is recorded once, no matter how many constraints the table has. + +Readers must determine the status of a constraint for a snapshot as follows: -Writers must record a status for every constraint that is active when a snapshot is created: +1. If the snapshot has no `constraint-statuses`, the snapshot makes no claim about any constraint +2. If the constraint's `constraint-id` is greater than the snapshot's `last-constraint-id`, the constraint did not exist when the snapshot was created and the snapshot makes no claim about it +3. If the constraint is listed in `statuses`, its status is the listed status +4. Otherwise, its status is `default-status` -* `validated` must not be recorded unless the constraint was checked for every row in the snapshot -* `valid` must not be recorded unless the constraint was enforced for the commit and the parent snapshot's status for the constraint is `validated` or `valid` -* `invalid` must not be recorded unless a row in the snapshot is known to violate the constraint -* `unvalidated` must be recorded when no other status can be recorded +`last-constraint-id` is what makes a defaulted status safe. Without it, a `default-status` recorded before a constraint was added would apply to that constraint, and a reader could conclude that a constraint held for a snapshot created before the constraint existed. -Enforcing a constraint for a commit is not sufficient to record `valid`. When the parent snapshot's status is not `validated` or `valid`, rows added by earlier commits were never checked, so the status is `unvalidated` even though the writer verified the rows that it added. +Writers must record `constraint-statuses` in every snapshot of a table that has constraints, must set `last-constraint-id` to the table's `last-constraint-id` when the snapshot is created, and must choose `default-status` and `statuses` so that every constraint resolves to a status that follows these rules: + +* A constraint must not resolve to `validated` unless it was checked for every row in the snapshot +* A constraint must not resolve to `valid` unless it was enforced for the commit and the parent snapshot's status for the constraint is `validated` or `valid` +* A constraint must not resolve to `invalid` unless a row in the snapshot is known to violate it +* `unvalidated` is the status of a constraint that cannot resolve to any other status + +Writers should choose the `default-status` that produces the shortest `statuses` list. + +Enforcing a constraint for a commit is not sufficient to resolve to `valid`. When the parent snapshot's status is not `validated` or `valid`, rows added by earlier commits were never checked, so the status is `unvalidated` even though the writer verified the rows that it added. When a constraint becomes enforced, either by being added with `enforced` set to true or by `enforced` changing from false to true, writers should validate the table and record `validated`. A writer that does not validate records `unvalidated`, and the constraint remains `unvalidated` until a later validation records `validated`. +A snapshot's `constraint-statuses` must not be modified after the snapshot is created. Recording a different status for a constraint requires a new snapshot. A snapshot that changes only constraint statuses may reuse its parent's manifest list. + +A constraint that a snapshot states holds may later be found not to hold for that snapshot, because `valid` relies on writers rather than on checking data. Writers should then commit a snapshot that records `invalid` and should expire the snapshots that state that the constraint holds, because queries against those snapshots would otherwise continue to rely on a constraint that does not hold. + Only `validated` and `valid` state that a constraint holds. They are distinguished so that readers can tell whether that conclusion was reached by checking data or by relying on writers to enforce the constraint, and so that the most recent `validated` snapshot can be found if a constraint is later found to be `invalid`. Writers may commit to a table where a constraint is `invalid`. An enforced constraint requires that a writer not add rows that violate the constraint; it does not require a writer to repair existing violations. @@ -1076,7 +1099,7 @@ A snapshot consists of the following fields: | _required_ | **`first-row-id`** | The first `_row_id` assigned to the first row in the first data file in the first manifest, see [Row Lineage](#row-lineage) | | _required_ | **`added-rows`** | The upper bound of the number of rows with assigned row IDs, see [Row Lineage](#row-lineage) | | _optional_ | **`key-id`** | ID of the encryption key that encrypts the manifest list key metadata | - | _optional_ | **`constraint-statuses`** | A list (optional) of constraint statuses for the snapshot, see [Constraint Validation](#constraint-validation) | + | _optional_ | **`constraint-statuses`** | A struct (optional) that records the status of the table's constraints for the snapshot, see [Constraint Validation](#constraint-validation) | The snapshot summary's `operation` field is used by some operations, like snapshot expiration, to skip processing certain snapshots. Possible `operation` values are: From 02014ff0becf9b25cc15c1283f293b93834ca7ef Mon Sep 17 00:00:00 2001 From: Huaxin Gao Date: Tue, 1 Sep 2026 18:48:28 -0700 Subject: [PATCH 04/25] fix constraint spec based on the sync --- format/spec.md | 74 ++++++++++++++++++++------------------------------ 1 file changed, 30 insertions(+), 44 deletions(-) diff --git a/format/spec.md b/format/spec.md index 394a4001c3bd..455dbada90a5 100644 --- a/format/spec.md +++ b/format/spec.md @@ -666,7 +666,7 @@ Constraints are added in v4 and are not supported in v3 or earlier. Three constraint types are defined: * `check` -- every row must satisfy a predicate -* `unique` -- the values of a set of fields must be distinct across all rows; a null value is not equal to any other null value, so more than one row may be null +* `unique` -- the non-null values of a set of fields must be distinct across all rows; more than one row may have a null value * `primary-key` -- the values of a set of fields must be distinct across all rows and must not be null Constraints are stored separately from schemas because the two evolve independently. Every constraint references the fields that it applies to by field ID, so a constraint continues to apply to the same columns after a column is renamed or reordered. @@ -684,21 +684,10 @@ A constraint consists of the following fields: | _required_ | **`name`** | `string` | A name for the constraint that is unique within the table. Names are for human consumption and must not be used to identify a constraint in metadata | | _required_ | **`enforced`** | `boolean` | Whether writers must verify that the rows they add satisfy the constraint | | _required_ | **`timestamp-ms`** | `long` | Timestamp in milliseconds from the unix epoch when the constraint was created or last modified. The timestamp is informational and must not be used to determine whether a constraint applies to a snapshot or whether it holds | -| _required_ | **`constraint-metadata`** | `struct` | Defines what the constraint requires; its fields depend on the constraint `type` (see below) | +| _optional_ | **`expression`** | `expression` | The predicate that every row must satisfy, see [Check Constraint Expressions](#check-constraint-expressions). Required for a `check` constraint and must not be set for other types | +| _optional_ | **`field-ids`** | `list` | The list of field IDs that the constraint applies to. Required for a `unique` or `primary-key` constraint and must not be set for a `check` constraint | -`constraint-metadata` for a `check` constraint consists of: - -| Requirement | Field name | Type | Description | -|-------------|------------------|--------------|-------------| -| _required_ | **`expression`** | `expression` | A predicate that every row must satisfy, see [Check Constraint Expressions](#check-constraint-expressions) | - -`constraint-metadata` for a `unique` or `primary-key` constraint consists of: - -| Requirement | Field name | Type | Description | -|-------------|-----------------|-------------|-------------| -| _required_ | **`field-ids`** | `list` | A list of field IDs that the constraint applies to | - -Each type carries only the metadata that it requires: a `check` constraint must not declare `field-ids` and a `unique` or `primary-key` constraint must not declare an `expression`. This keeps a single source of truth for the fields that a constraint references. +The fields that define what a constraint requires are embedded directly in the constraint based on its `type`. Each type carries only the metadata that it requires: a `check` constraint has an `expression` and must not declare `field-ids`, and a `unique` or `primary-key` constraint has `field-ids` and must not declare an `expression`. This keeps a single source of truth for the fields that a constraint references. The `field-ids` of a `unique` or `primary-key` constraint must reference primitive fields that are either top-level fields or nested in required structs, and must not reference fields within a `list` or a `map`. These are the same restrictions that apply to [identifier fields](#identifier-field-ids). @@ -706,6 +695,12 @@ When a constraint is `enforced`, writers must verify that the rows they add sati Whether to trust a constraint that is not enforced is left to engines and is not tracked in table metadata. +A table may have at most one `primary-key` constraint. A key that spans several fields is expressed as a single `primary-key` constraint over multiple `field-ids`. + +A `primary-key` constraint replaces [identifier field IDs](#identifier-field-ids), which express the same concept: a set of fields that identifies a row, without a uniqueness guarantee. When a table is upgraded to v4, its `identifier-field-ids` are rewritten as a `primary-key` constraint that is not enforced. Identifier field IDs are not used in v4. + +Only a constraint's `name` and `enforced` fields may be changed in place. Changing the `expression` of a `check` constraint or the `field-ids` of a `unique` or `primary-key` constraint changes what the constraint requires, so it must be done by removing the constraint and adding a new one with a new `constraint-id`, so that statuses recorded for the old definition are not read as applying to the new one. + Constraint IDs are assigned from the table's `last-constraint-id`, which is treated as 0 when it is not present. Writers must assign a new constraint an ID that is higher than the table's current `last-constraint-id` and must update `last-constraint-id` to the highest assigned ID. Constraint IDs must not be reused after the constraint that used an ID is removed, because retained snapshots may still reference the removed ID. Readers must not assume that every `constraint-id` referenced by a snapshot is present in `constraints`. #### Check Constraint Expressions @@ -731,42 +726,33 @@ The status of a constraint for a snapshot is one of: | `invalid` | The constraint was checked and at least one row in the snapshot violates it | | `unvalidated` | Whether the constraint holds for all rows in the snapshot is not known | -A snapshot's `constraint-statuses` consists of: - -| Requirement | Field name | Type | Description | -|-------------|--------------------------|----------------|-------------| -| _required_ | **`last-constraint-id`** | `int` | The table's `last-constraint-id` when the snapshot was created | -| _required_ | **`default-status`** | `string` | The status of every constraint that is not listed in `statuses` | -| _optional_ | **`statuses`** | `list` | Statuses for constraints whose status is not `default-status` | +A snapshot's `constraint-statuses` records, for each status, the IDs of the constraints that have that status for the snapshot: -Each entry of `statuses` consists of: +| Requirement | Field name | Type | Description | +|-------------|-------------------|-------------|-------------| +| _optional_ | **`validated`** | `list` | IDs of constraints that are `validated` for the snapshot | +| _optional_ | **`valid`** | `list` | IDs of constraints that are `valid` for the snapshot | +| _optional_ | **`invalid`** | `list` | IDs of constraints that are `invalid` for the snapshot | +| _optional_ | **`unvalidated`** | `list` | IDs of constraints that are `unvalidated` for the snapshot | -| Requirement | Field name | Type | Description | -|-------------|---------------------|----------|-------------| -| _required_ | **`constraint-id`** | `int` | ID of the constraint that the status describes | -| _required_ | **`status`** | `string` | The status of the constraint for this snapshot | +Each list contains the `constraint-id` of every constraint that has that status for the snapshot. A `constraint-id` must appear in at most one list. A list with no constraints may be omitted. A constraint whose ID is not present in any list makes no claim for the snapshot and is treated as `unvalidated`. -`default-status` covers the constraints that share a status and `statuses` records only the constraints that differ from it, so the size of `constraint-statuses` does not grow with the number of constraints in a table. When every constraint has the same status, that status is recorded once, no matter how many constraints the table has. +This is an explicit representation: each constraint's status is recorded independently, so the size of `constraint-statuses` grows with the number of constraints in a table. This keeps the encoding simple; more compact representations may be added in a later version if it becomes a problem. Readers must determine the status of a constraint for a snapshot as follows: 1. If the snapshot has no `constraint-statuses`, the snapshot makes no claim about any constraint -2. If the constraint's `constraint-id` is greater than the snapshot's `last-constraint-id`, the constraint did not exist when the snapshot was created and the snapshot makes no claim about it -3. If the constraint is listed in `statuses`, its status is the listed status -4. Otherwise, its status is `default-status` - -`last-constraint-id` is what makes a defaulted status safe. Without it, a `default-status` recorded before a constraint was added would apply to that constraint, and a reader could conclude that a constraint held for a snapshot created before the constraint existed. - -Writers must record `constraint-statuses` in every snapshot of a table that has constraints, must set `last-constraint-id` to the table's `last-constraint-id` when the snapshot is created, and must choose `default-status` and `statuses` so that every constraint resolves to a status that follows these rules: +2. If the constraint's `constraint-id` is listed in `validated`, `valid`, `invalid`, or `unvalidated`, that is its status +3. Otherwise, the snapshot makes no claim about the constraint and its status is treated as `unvalidated` -* A constraint must not resolve to `validated` unless it was checked for every row in the snapshot -* A constraint must not resolve to `valid` unless it was enforced for the commit and the parent snapshot's status for the constraint is `validated` or `valid` -* A constraint must not resolve to `invalid` unless a row in the snapshot is known to violate it -* `unvalidated` is the status of a constraint that cannot resolve to any other status +Writers must record `constraint-statuses` in every snapshot of a table that has constraints, and must place each constraint's ID in the list for a status that follows these rules: -Writers should choose the `default-status` that produces the shortest `statuses` list. +* A constraint must not be listed as `validated` unless it was checked for every row in the snapshot +* A constraint must not be listed as `valid` unless it was enforced for the commit and the parent snapshot's status for the constraint is `validated` or `valid` +* A constraint must not be listed as `invalid` unless a row in the snapshot is known to violate it +* `unvalidated` is the status of a constraint that cannot be listed in any other status -Enforcing a constraint for a commit is not sufficient to resolve to `valid`. When the parent snapshot's status is not `validated` or `valid`, rows added by earlier commits were never checked, so the status is `unvalidated` even though the writer verified the rows that it added. +Enforcing a constraint for a commit is not sufficient to list it as `valid`. When the parent snapshot's status is not `validated` or `valid`, rows added by earlier commits were never checked, so the status is `unvalidated` even though the writer verified the rows that it added. When a constraint becomes enforced, either by being added with `enforced` set to true or by `enforced` changing from false to true, writers should validate the table and record `validated`. A writer that does not validate records `unvalidated`, and the constraint remains `unvalidated` until a later validation records `validated`. @@ -1929,7 +1915,7 @@ A metadata JSON file may be compressed with [GZIP](https://datatracker.ietf.org/ |**`last-partition-id`**|`JSON int`|`1000`| |**`properties`**|`JSON object: {`
  `"": "",`
  `...`
`}`|`{`
  `"write.format.default": "avro",`
  `"commit.retry.num-retries": "4"`
`}`| |**`current-snapshot-id`**|`JSON long`|`3051729675574597004`| -|**`snapshots`**|`JSON list of objects: [ {`
  `"snapshot-id": ,`
  `"timestamp-ms": ,`
  `"summary": {`
    `"operation": ,`
    `... },`
  `"manifest-list": "",`
  `"schema-id": ""`
  `},`
  `...`
`]`|`[ {`
  `"snapshot-id": 3051729675574597004,`
  `"timestamp-ms": 1515100955770,`
  `"summary": {`
    `"operation": "append"`
  `},`
  `"manifest-list": "s3://b/wh/.../s1.avro"`
  `"schema-id": 0`
`} ]`| +|**`snapshots`**|`JSON list of objects: [ {`
  `"snapshot-id": ,`
  `"timestamp-ms": ,`
  `"summary": {`
    `"operation": ,`
    `... },`
  `"manifest-list": "",`
  `"schema-id": "",`
  `"constraint-statuses": {`
    `"validated": [ ... ],`
    `... }`
  `},`
  `...`
`]`|`[ {`
  `"snapshot-id": 3051729675574597004,`
  `"timestamp-ms": 1515100955770,`
  `"summary": {`
    `"operation": "append"`
  `},`
  `"manifest-list": "s3://b/wh/.../s1.avro"`
  `"schema-id": 0,`
  `"constraint-statuses": {`
    `"validated": [1],`
    `"unvalidated": [2]`
  `}`
`} ]`| |**`snapshot-log`**|`JSON list of objects: [`
  `{`
  `"snapshot-id": ,`
  `"timestamp-ms":`
  `},`
  `...`
`]`|`[ {`
  `"snapshot-id": 30517296...,`
  `"timestamp-ms": 1515100...`
`} ]`| |**`metadata-log`**|`JSON list of objects: [`
  `{`
  `"metadata-file": ,`
  `"timestamp-ms":`
  `},`
  `...`
`]`|`[ {`
  `"metadata-file": "s3://bucket/.../v1.json",`
  `"timestamp-ms": 1515100...`
`} ]` | |**`sort-orders`**|`JSON sort orders (list of sort field object)`|`See above`| @@ -1937,7 +1923,7 @@ A metadata JSON file may be compressed with [GZIP](https://datatracker.ietf.org/ |**`refs`**|`JSON map with string key and object value:`
`{`
  `"": {`
  `"snapshot-id": ,`
  `"type": ,`
  `"max-ref-age-ms": ,`
  `...`
  `}`
  `...`
`}`|`{`
  `"test": {`
  `"snapshot-id": 123456789000,`
  `"type": "tag",`
  `"max-ref-age-ms": 10000000`
  `}`
`}`| |**`encryption-keys`**|`JSON list of encryption key objects`|`[ {"key-id": "5f819b", "key-metadata": "aWNlYmVyZwo="} ]`| |**`last-constraint-id`**|`JSON int`|`2`| -|**`constraints`**|`JSON list of objects: [ {`
  `"constraint-id": ,`
  `"type": ,`
  `"name": "",`
  `"enforced": ,`
  `"timestamp-ms": ,`
  `"constraint-metadata": { ... }`
  `},`
  `...`
`]`|`[ {`
  `"constraint-id": 1,`
  `"type": "primary-key",`
  `"name": "pk_sales",`
  `"enforced": false,`
  `"timestamp-ms": 1515100955770,`
  `"constraint-metadata": {`
    `"field-ids": [1]`
  `}`
`} ]`| +|**`constraints`**|`JSON list of objects: [ {`
  `"constraint-id": ,`
  `"type": "check",`
  `"name": "",`
  `"enforced": ,`
  `"timestamp-ms": ,`
  `"expression": { ... }`
`}, {`
  `"constraint-id": ,`
  `"type": "primary-key",`
  `"name": "",`
  `"enforced": ,`
  `"timestamp-ms": ,`
  `"field-ids": [ ... ]`
`},`
  `...`
`]`|`[ {`
  `"constraint-id": 1,`
  `"type": "check",`
  `"name": "chk_amount",`
  `"enforced": true,`
  `"timestamp-ms": 1515100955770,`
  `"expression": { ... }`
`}, {`
  `"constraint-id": 2,`
  `"type": "primary-key",`
  `"name": "pk_sales",`
  `"enforced": false,`
  `"timestamp-ms": 1515100955770,`
  `"field-ids": [1]`
`} ]`| ### Name Mapping Serialization @@ -2060,7 +2046,7 @@ Constraints are added in v4: * [Constraints](#constraints) must not be added to v3 or earlier tables * Table metadata may contain `constraints` and `last-constraint-id` * Snapshots may contain `constraint-statuses` -* Upgrading a v2 or v3 table to v4 does not add constraints; a table upgraded to v4 has no constraints until they are added +* Upgrading a v2 or v3 table to v4 rewrites the table's `identifier-field-ids` as a `primary-key` constraint that is not enforced; a table with no identifier fields has no constraints until they are added ### Version 3 From d442b647aa2c81486d71aa20592de88fb47f2cc1 Mon Sep 17 00:00:00 2001 From: Huaxin Gao Date: Tue, 1 Sep 2026 19:08:49 -0700 Subject: [PATCH 05/25] address comments --- format/spec.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/format/spec.md b/format/spec.md index 455dbada90a5..7a04a7b57e08 100644 --- a/format/spec.md +++ b/format/spec.md @@ -691,6 +691,8 @@ The fields that define what a constraint requires are embedded directly in the c The `field-ids` of a `unique` or `primary-key` constraint must reference primitive fields that are either top-level fields or nested in required structs, and must not reference fields within a `list` or a `map`. These are the same restrictions that apply to [identifier fields](#identifier-field-ids). +A field referenced by a constraint must not be dropped. A writer must reject a schema change that removes a field referenced by a `check` constraint's `expression` or by a `unique` or `primary-key` constraint's `field-ids` unless the constraint is removed in the same change. + When a constraint is `enforced`, writers must verify that the rows they add satisfy the constraint and must fail the write if they do not. A writer that cannot verify an enforced constraint must reject writes to the table rather than add rows that have not been verified. When a constraint is not enforced, writers are not required to verify the rows they add. Whether to trust a constraint that is not enforced is left to engines and is not tracked in table metadata. @@ -707,7 +709,7 @@ Constraint IDs are assigned from the table's `last-constraint-id`, which is trea The `expression` of a `check` constraint is serialized as described in the [Iceberg expressions spec](expressions-spec.md) and must use ID references so that it remains bound to the same fields when columns are renamed or reordered. -A check expression must be a deterministic predicate over a single row. Expressions that depend on more than one row, such as aggregates and window functions, and expressions that depend on another table, such as subqueries, must not be used. +A check expression is evaluated for each row over the values of that row. Expressions that depend on more than one row, such as aggregates and window functions, and expressions that depend on another table, such as subqueries, must not be used. Iceberg predicates use two-valued logic: a predicate always produces true or false and never produces null, so a comparison with a null operand produces false. This differs from SQL `CHECK`, where a row satisfies a constraint unless the predicate produces false and a null value therefore satisfies the constraint. From 724fd32e9bd373834db4ea00a449dc773fb18b48 Mon Sep 17 00:00:00 2001 From: Huaxin Gao Date: Sun, 6 Sep 2026 19:17:26 -0700 Subject: [PATCH 06/25] Spec: add constraints and schema evolution section --- format/spec.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/format/spec.md b/format/spec.md index 7a04a7b57e08..9d690083c0ee 100644 --- a/format/spec.md +++ b/format/spec.md @@ -691,8 +691,6 @@ The fields that define what a constraint requires are embedded directly in the c The `field-ids` of a `unique` or `primary-key` constraint must reference primitive fields that are either top-level fields or nested in required structs, and must not reference fields within a `list` or a `map`. These are the same restrictions that apply to [identifier fields](#identifier-field-ids). -A field referenced by a constraint must not be dropped. A writer must reject a schema change that removes a field referenced by a `check` constraint's `expression` or by a `unique` or `primary-key` constraint's `field-ids` unless the constraint is removed in the same change. - When a constraint is `enforced`, writers must verify that the rows they add satisfy the constraint and must fail the write if they do not. A writer that cannot verify an enforced constraint must reject writes to the table rather than add rows that have not been verified. When a constraint is not enforced, writers are not required to verify the rows they add. Whether to trust a constraint that is not enforced is left to engines and is not tracked in table metadata. @@ -715,6 +713,16 @@ Iceberg predicates use two-valued logic: a predicate always produces true or fal To express SQL `CHECK` semantics for an optional field, the stored expression must make the null case explicit. For example, SQL `CHECK (price >= 0)` for an optional `price` field is stored as the expression for `price >= 0 OR price IS NULL`. This is unnecessary for required fields, which can never be null. +#### Constraints and Schema Evolution + +A constraint references fields by ID, so schema changes interact with constraints as follows. The referenced fields of a `check` constraint are the field IDs in its `expression`; the referenced fields of a `unique` or `primary-key` constraint are its `field-ids`. + +* Renaming or reordering a referenced field is allowed; the constraint continues to apply to the same fields. +* If a dropped field is referenced only by single-column constraints, the drop is allowed and those constraints are removed automatically. If a dropped field is referenced by a multi-column constraint, the writer must reject the drop unless that constraint is removed in the same change. +* The type of a referenced field must not be changed, even for type promotions that are otherwise allowed. This restriction may be relaxed in a later version. + +Changing a constraint's own definition is governed by the mutability rule above: only `name` and `enforced` may be changed in place; changing an `expression` or `field-ids` requires a new constraint. + #### Constraint Validation Enforcement and validation are separate properties. Whether writers must verify the rows that they add is a property of a constraint, tracked by `enforced`. Whether a table is known to satisfy a constraint is a property of a table's data, tracked per snapshot by `constraint-statuses`. From 841e7a00982bd6ca67f5203ef14d6bbf65e46743 Mon Sep 17 00:00:00 2001 From: Huaxin Gao Date: Sun, 6 Sep 2026 19:53:12 -0700 Subject: [PATCH 07/25] clarify constraint status semantics --- format/spec.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/format/spec.md b/format/spec.md index 9d690083c0ee..4db376579334 100644 --- a/format/spec.md +++ b/format/spec.md @@ -745,7 +745,7 @@ A snapshot's `constraint-statuses` records, for each status, the IDs of the cons | _optional_ | **`invalid`** | `list` | IDs of constraints that are `invalid` for the snapshot | | _optional_ | **`unvalidated`** | `list` | IDs of constraints that are `unvalidated` for the snapshot | -Each list contains the `constraint-id` of every constraint that has that status for the snapshot. A `constraint-id` must appear in at most one list. A list with no constraints may be omitted. A constraint whose ID is not present in any list makes no claim for the snapshot and is treated as `unvalidated`. +Each list contains the `constraint-id` of every constraint that has that status for the snapshot. Every constraint that exists when the snapshot is created must be listed in exactly one of the four lists, and a `constraint-id` must not appear in more than one list. A list with no constraints may be omitted. A constraint whose ID is not present in any list did not exist when the snapshot was created, so the snapshot makes no claim about it. This is an explicit representation: each constraint's status is recorded independently, so the size of `constraint-statuses` grows with the number of constraints in a table. This keeps the encoding simple; more compact representations may be added in a later version if it becomes a problem. @@ -753,9 +753,9 @@ Readers must determine the status of a constraint for a snapshot as follows: 1. If the snapshot has no `constraint-statuses`, the snapshot makes no claim about any constraint 2. If the constraint's `constraint-id` is listed in `validated`, `valid`, `invalid`, or `unvalidated`, that is its status -3. Otherwise, the snapshot makes no claim about the constraint and its status is treated as `unvalidated` +3. Otherwise, the constraint did not exist when the snapshot was created and the snapshot makes no claim about it -Writers must record `constraint-statuses` in every snapshot of a table that has constraints, and must place each constraint's ID in the list for a status that follows these rules: +Writers must record `constraint-statuses` in every snapshot of a table that has constraints, and must place every constraint that exists when the snapshot is created into exactly one status list, following these rules: * A constraint must not be listed as `validated` unless it was checked for every row in the snapshot * A constraint must not be listed as `valid` unless it was enforced for the commit and the parent snapshot's status for the constraint is `validated` or `valid` From 9fa1b4ce2c563b95675e2b7a8b34a81fa6c7a7f3 Mon Sep 17 00:00:00 2001 From: Huaxin Gao Date: Sun, 6 Sep 2026 21:56:54 -0700 Subject: [PATCH 08/25] clarify constraint validation status semantics --- format/spec.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/format/spec.md b/format/spec.md index 4db376579334..c3ec2ee8fdd9 100644 --- a/format/spec.md +++ b/format/spec.md @@ -768,7 +768,9 @@ When a constraint becomes enforced, either by being added with `enforced` set to A snapshot's `constraint-statuses` must not be modified after the snapshot is created. Recording a different status for a constraint requires a new snapshot. A snapshot that changes only constraint statuses may reuse its parent's manifest list. -A constraint that a snapshot states holds may later be found not to hold for that snapshot, because `valid` relies on writers rather than on checking data. Writers should then commit a snapshot that records `invalid` and should expire the snapshots that state that the constraint holds, because queries against those snapshots would otherwise continue to rely on a constraint that does not hold. +A constraint that a snapshot reports as `valid` may later be found not to hold for that snapshot. A `valid` status depends on every commit in the snapshot's history having correctly enforced the constraint, and Iceberg records the status a writer reports without re-checking the data. So if any of those writers was buggy or non-compliant, the `valid` status can be wrong even though nothing detected it at commit time. A `validated` status, which reflects an actual scan of all rows, does not depend on that chain. Writers should then commit a snapshot that records `invalid` and should expire the snapshots that state that the constraint holds, because queries against those snapshots would otherwise continue to rely on a constraint that does not hold. + +Iceberg does not detect violations on its own. A constraint is only found to be `invalid` when an engine scans the data and checks the constraint, for example during an explicit validation or when validating a newly enforced constraint, and then records `invalid`. Until such a scan runs, a `valid` status that does not actually hold is not detected. Only `validated` and `valid` state that a constraint holds. They are distinguished so that readers can tell whether that conclusion was reached by checking data or by relying on writers to enforce the constraint, and so that the most recent `validated` snapshot can be found if a constraint is later found to be `invalid`. From fdd5e3f961ed6bd5c2484af4c57568cbaa68ea76 Mon Sep 17 00:00:00 2001 From: Huaxin Gao Date: Mon, 14 Sep 2026 13:11:36 -0700 Subject: [PATCH 09/25] clarify check expression scope and incremental validation --- format/spec.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/format/spec.md b/format/spec.md index c3ec2ee8fdd9..cce05b7df5a4 100644 --- a/format/spec.md +++ b/format/spec.md @@ -707,7 +707,7 @@ Constraint IDs are assigned from the table's `last-constraint-id`, which is trea The `expression` of a `check` constraint is serialized as described in the [Iceberg expressions spec](expressions-spec.md) and must use ID references so that it remains bound to the same fields when columns are renamed or reordered. -A check expression is evaluated for each row over the values of that row. Expressions that depend on more than one row, such as aggregates and window functions, and expressions that depend on another table, such as subqueries, must not be used. +A check expression is evaluated for each row over the values of that row. An expression may reference more than one field of the row, such as `start_date <= end_date`. Expressions that depend on more than one row, such as aggregates and window functions, and expressions that depend on another table, such as subqueries, must not be used. Iceberg predicates use two-valued logic: a predicate always produces true or false and never produces null, so a comparison with a null operand produces false. This differs from SQL `CHECK`, where a row satisfies a constraint unless the predicate produces false and a null value therefore satisfies the constraint. @@ -766,6 +766,8 @@ Enforcing a constraint for a commit is not sufficient to list it as `valid`. Whe When a constraint becomes enforced, either by being added with `enforced` set to true or by `enforced` changing from false to true, writers should validate the table and record `validated`. A writer that does not validate records `unvalidated`, and the constraint remains `unvalidated` until a later validation records `validated`. +A writer does not have to check every row in a single scan. After checking every row in an ancestor snapshot, a writer may check only the rows added between that ancestor and the current snapshot and record `validated` for the current snapshot. This allows a validation to finish on a table that is written concurrently, without blocking writes or restarting the scan. + A snapshot's `constraint-statuses` must not be modified after the snapshot is created. Recording a different status for a constraint requires a new snapshot. A snapshot that changes only constraint statuses may reuse its parent's manifest list. A constraint that a snapshot reports as `valid` may later be found not to hold for that snapshot. A `valid` status depends on every commit in the snapshot's history having correctly enforced the constraint, and Iceberg records the status a writer reports without re-checking the data. So if any of those writers was buggy or non-compliant, the `valid` status can be wrong even though nothing detected it at commit time. A `validated` status, which reflects an actual scan of all rows, does not depend on that chain. Writers should then commit a snapshot that records `invalid` and should expire the snapshots that state that the constraint holds, because queries against those snapshots would otherwise continue to rely on a constraint that does not hold. From d4a899d763397c6b3e7ebd2a33a1cd1a7ca5b470 Mon Sep 17 00:00:00 2001 From: Huaxin Gao Date: Fri, 25 Sep 2026 13:35:00 -0700 Subject: [PATCH 10/25] address comments --- format/spec.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/format/spec.md b/format/spec.md index cce05b7df5a4..9926a0c2b811 100644 --- a/format/spec.md +++ b/format/spec.md @@ -657,21 +657,19 @@ A data or delete file is associated with a sort order by the sort order's id wit ### Constraints +Constraints are added in v4 and are not supported in v3 or earlier. + A **constraint** declares a property that a table's rows are expected to satisfy. A constraint's definition is stored in table metadata. Whether a constraint holds is recorded for each snapshot, see [Constraint Validation](#constraint-validation). Iceberg does not evaluate constraints. Enforcement and validation are performed by engines that write to a table. Iceberg stores constraint definitions and records the status that a writer reports for a commit without verifying it. -Constraints are added in v4 and are not supported in v3 or earlier. - Three constraint types are defined: * `check` -- every row must satisfy a predicate * `unique` -- the non-null values of a set of fields must be distinct across all rows; more than one row may have a null value * `primary-key` -- the values of a set of fields must be distinct across all rows and must not be null -Constraints are stored separately from schemas because the two evolve independently. Every constraint references the fields that it applies to by field ID, so a constraint continues to apply to the same columns after a column is renamed or reordered. - -A required field in a schema expresses `NOT NULL`. It is not represented as a constraint. +Constraints are stored separately from schemas because they span multiple fields and evolve independently. Every constraint references the fields that it applies to by field ID, so a constraint continues to apply to the same columns after a column is renamed or reordered. #### Constraint Fields From 195121960df1b8611a8108e866ce93c68ff9eb6d Mon Sep 17 00:00:00 2001 From: Huaxin Gao Date: Fri, 25 Sep 2026 17:08:54 -0700 Subject: [PATCH 11/25] address comments --- format/spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/format/spec.md b/format/spec.md index 9926a0c2b811..cf343a053c7e 100644 --- a/format/spec.md +++ b/format/spec.md @@ -689,7 +689,7 @@ The fields that define what a constraint requires are embedded directly in the c The `field-ids` of a `unique` or `primary-key` constraint must reference primitive fields that are either top-level fields or nested in required structs, and must not reference fields within a `list` or a `map`. These are the same restrictions that apply to [identifier fields](#identifier-field-ids). -When a constraint is `enforced`, writers must verify that the rows they add satisfy the constraint and must fail the write if they do not. A writer that cannot verify an enforced constraint must reject writes to the table rather than add rows that have not been verified. When a constraint is not enforced, writers are not required to verify the rows they add. +When a constraint is `enforced`, writers must verify that the rows they add satisfy the constraint and must fail the write if they do not. A writer that cannot verify an enforced constraint must reject writes to the table. When a constraint is not enforced, writers are not required to verify the rows they add. Whether to trust a constraint that is not enforced is left to engines and is not tracked in table metadata. From 61962533f41e6f5dd3315162811b1f81dc2e0651 Mon Sep 17 00:00:00 2001 From: Huaxin Gao Date: Tue, 29 Sep 2026 17:47:16 -0700 Subject: [PATCH 12/25] address comments --- format/spec.md | 66 ++++++++++++++++++++++++++------------------------ 1 file changed, 34 insertions(+), 32 deletions(-) diff --git a/format/spec.md b/format/spec.md index cf343a053c7e..531271f30884 100644 --- a/format/spec.md +++ b/format/spec.md @@ -436,6 +436,8 @@ Two rows are the "same"---that is, the rows represent the same entity---if the i Identifier fields may be nested in structs but cannot be nested within maps or lists. Float, double, and optional fields cannot be used as identifier fields and a nested field cannot be used as an identifier field if it is nested in an optional struct, to avoid null values in identifiers. +Identifier field IDs are not used in v4, where a [`primary-key` constraint](#constraints) expresses the same concept. + #### Reserved Field IDs Iceberg tables must not use field ids greater than 2147483447 (`Integer.MAX_VALUE - 200`). This id range is reserved for metadata columns that can be used in user data schemas, like the `_file` column that holds the file path in which a row was stored. @@ -659,17 +661,15 @@ A data or delete file is associated with a sort order by the sort order's id wit Constraints are added in v4 and are not supported in v3 or earlier. -A **constraint** declares a property that a table's rows are expected to satisfy. A constraint's definition is stored in table metadata. Whether a constraint holds is recorded for each snapshot, see [Constraint Validation](#constraint-validation). - -Iceberg does not evaluate constraints. Enforcement and validation are performed by engines that write to a table. Iceberg stores constraint definitions and records the status that a writer reports for a commit without verifying it. +A **constraint** declares a condition that a table's rows are expected to satisfy. Its definition is stored in table metadata. Each snapshot contains the writer-reported status of each constraint that existed when the snapshot was created; see [Constraint Validation](#constraint-validation). Three constraint types are defined: * `check` -- every row must satisfy a predicate -* `unique` -- the non-null values of a set of fields must be distinct across all rows; more than one row may have a null value +* `unique` -- the values of a set of fields must be distinct across the rows in which every field in the set is non-null; a row in which any field in the set is null is not compared, so more than one such row may exist * `primary-key` -- the values of a set of fields must be distinct across all rows and must not be null -Constraints are stored separately from schemas because they span multiple fields and evolve independently. Every constraint references the fields that it applies to by field ID, so a constraint continues to apply to the same columns after a column is renamed or reordered. +A constraint is stored separately from the table schema and must refer to fields by field ID. Constraints may evolve independently from the schema, being added, modified, or removed. #### Constraint Fields @@ -680,24 +680,26 @@ A constraint consists of the following fields: | _required_ | **`constraint-id`** | `int` | ID of the constraint; unique within the table | | _required_ | **`type`** | `string` | The constraint type: `check`, `unique`, or `primary-key` | | _required_ | **`name`** | `string` | A name for the constraint that is unique within the table. Names are for human consumption and must not be used to identify a constraint in metadata | -| _required_ | **`enforced`** | `boolean` | Whether writers must verify that the rows they add satisfy the constraint | +| _required_ | **`enforced`** | `boolean` | Whether writers must only add rows that satisfy the constraint | | _required_ | **`timestamp-ms`** | `long` | Timestamp in milliseconds from the unix epoch when the constraint was created or last modified. The timestamp is informational and must not be used to determine whether a constraint applies to a snapshot or whether it holds | | _optional_ | **`expression`** | `expression` | The predicate that every row must satisfy, see [Check Constraint Expressions](#check-constraint-expressions). Required for a `check` constraint and must not be set for other types | | _optional_ | **`field-ids`** | `list` | The list of field IDs that the constraint applies to. Required for a `unique` or `primary-key` constraint and must not be set for a `check` constraint | The fields that define what a constraint requires are embedded directly in the constraint based on its `type`. Each type carries only the metadata that it requires: a `check` constraint has an `expression` and must not declare `field-ids`, and a `unique` or `primary-key` constraint has `field-ids` and must not declare an `expression`. This keeps a single source of truth for the fields that a constraint references. -The `field-ids` of a `unique` or `primary-key` constraint must reference primitive fields that are either top-level fields or nested in required structs, and must not reference fields within a `list` or a `map`. These are the same restrictions that apply to [identifier fields](#identifier-field-ids). +The `field-ids` of a `unique` or `primary-key` constraint must reference primitive fields and must not reference fields within a `list` or a `map`. Fields of type `float`, `double`, `geometry`, or `geography` must not be used because Iceberg does not define equality for them. Fields of type `unknown` must not be used because their values are always null. -When a constraint is `enforced`, writers must verify that the rows they add satisfy the constraint and must fail the write if they do not. A writer that cannot verify an enforced constraint must reject writes to the table. When a constraint is not enforced, writers are not required to verify the rows they add. +The fields of a `primary-key` constraint must be `required` and must not be nested in an optional struct, so that the schema rules out null keys even when the constraint is not enforced. These are the same restrictions that apply to [identifier fields](#identifier-field-ids). The fields of a `unique` constraint may be optional or nested in optional structs, because a row in which any field is null is not compared. -Whether to trust a constraint that is not enforced is left to engines and is not tracked in table metadata. +When a constraint is `enforced`, writers must only add rows they can prove follow the constraint. A constraint that is not enforced can be written to by any writer, regardless of their ability to prove that the constraint is followed by new rows. -A table may have at most one `primary-key` constraint. A key that spans several fields is expressed as a single `primary-key` constraint over multiple `field-ids`. +When a commit is retried against a new parent snapshot, the rows that a writer adds must still satisfy the table's enforced constraints. A `check` constraint is evaluated for each row on its own, so a retry cannot change its result. A `unique` or `primary-key` constraint compares rows to each other, so a writer must verify the rows that it adds against the new parent; verifying them against the original parent is not sufficient. A writer must also verify the rows that it adds against any constraint that a concurrent commit added or changed to enforced, and must recompute `constraint-statuses` from the new parent. -A `primary-key` constraint replaces [identifier field IDs](#identifier-field-ids), which express the same concept: a set of fields that identifies a row, without a uniqueness guarantee. When a table is upgraded to v4, its `identifier-field-ids` are rewritten as a `primary-key` constraint that is not enforced. Identifier field IDs are not used in v4. +Whether to trust a constraint that is not enforced is left to engines and is not tracked in table metadata. -Only a constraint's `name` and `enforced` fields may be changed in place. Changing the `expression` of a `check` constraint or the `field-ids` of a `unique` or `primary-key` constraint changes what the constraint requires, so it must be done by removing the constraint and adding a new one with a new `constraint-id`, so that statuses recorded for the old definition are not read as applying to the new one. +A table may have at most one `primary-key` constraint. A `primary-key` constraint replaces [identifier field IDs](#identifier-field-ids), which express the same concept: a set of fields that identifies a row, without a uniqueness guarantee. Identifier field IDs are not used in v4. + +When a table is upgraded to v4, the `identifier-field-ids` of the table's current schema are rewritten as a single `primary-key` constraint that is not enforced. The constraint is assigned a `constraint-id` from `last-constraint-id` in the same way as any other constraint, its `name` is `pk`, and its `timestamp-ms` is the time of the upgrade. A table whose current schema has no identifier fields has no constraints until they are added. Writers must not set `identifier-field-ids` in a schema that is added to a v4 table, and readers must ignore `identifier-field-ids` in a v4 table. Constraint IDs are assigned from the table's `last-constraint-id`, which is treated as 0 when it is not present. Writers must assign a new constraint an ID that is higher than the table's current `last-constraint-id` and must update `last-constraint-id` to the highest assigned ID. Constraint IDs must not be reused after the constraint that used an ID is removed, because retained snapshots may still reference the removed ID. Readers must not assume that every `constraint-id` referenced by a snapshot is present in `constraints`. @@ -707,7 +709,11 @@ The `expression` of a `check` constraint is serialized as described in the [Iceb A check expression is evaluated for each row over the values of that row. An expression may reference more than one field of the row, such as `start_date <= end_date`. Expressions that depend on more than one row, such as aggregates and window functions, and expressions that depend on another table, such as subqueries, must not be used. -Iceberg predicates use two-valued logic: a predicate always produces true or false and never produces null, so a comparison with a null operand produces false. This differs from SQL `CHECK`, where a row satisfies a constraint unless the predicate produces false and a null value therefore satisfies the constraint. +A check expression must produce the same result every time it is evaluated for the same row. A function that depends on anything other than its arguments, such as the current time or a random value, must not be called, because the status recorded for a snapshot describes the table's data and an expression whose result can change on its own would make a recorded status wrong without any write. A [user-defined function](udf-spec.md) records whether it is deterministic. + +Changing the definition of a function that a check expression calls changes what the constraint requires even though the constraint itself is unchanged. Statuses recorded before the change do not describe the new definition, so a writer that changes such a function should validate the constraint again. + +Iceberg predicates use two-valued logic and null-safe comparisons, as defined in the [expressions spec](expressions-spec.md#boolean-logic). This differs from SQL `CHECK`, where a row satisfies a constraint unless the predicate produces false. To express SQL `CHECK` semantics for an optional field, the stored expression must make the null case explicit. For example, SQL `CHECK (price >= 0)` for an optional `price` field is stored as the expression for `price >= 0 OR price IS NULL`. This is unnecessary for required fields, which can never be null. @@ -716,10 +722,11 @@ To express SQL `CHECK` semantics for an optional field, the stored expression mu A constraint references fields by ID, so schema changes interact with constraints as follows. The referenced fields of a `check` constraint are the field IDs in its `expression`; the referenced fields of a `unique` or `primary-key` constraint are its `field-ids`. * Renaming or reordering a referenced field is allowed; the constraint continues to apply to the same fields. -* If a dropped field is referenced only by single-column constraints, the drop is allowed and those constraints are removed automatically. If a dropped field is referenced by a multi-column constraint, the writer must reject the drop unless that constraint is removed in the same change. -* The type of a referenced field must not be changed, even for type promotions that are otherwise allowed. This restriction may be relaxed in a later version. +* Every field referenced by a constraint must exist in the table's current schema. A schema change that removes a referenced field must be rejected unless the constraint is removed in the same change. +* The type of a field referenced by a `check` constraint must not be changed, even for type promotions that are otherwise allowed, because the constraint's `expression` contains literals that are bound to the field's type. The type of a field referenced by a `unique` or `primary-key` constraint may be changed by a type promotion, which preserves values and therefore preserves distinctness. +* A field referenced by a `primary-key` constraint must not be made optional, and a struct that contains such a field must not be made optional, because a `primary-key` requires its fields to be non-null. -Changing a constraint's own definition is governed by the mutability rule above: only `name` and `enforced` may be changed in place; changing an `expression` or `field-ids` requires a new constraint. +Only a constraint's `name` and `enforced` fields may be changed in place, and a writer that changes either must update `timestamp-ms`. Changing the `expression` of a `check` constraint or the `field-ids` of a `unique` or `primary-key` constraint changes what the constraint requires, so it must be done by removing the constraint and adding a new one with a new `constraint-id`, so that statuses recorded for the old definition are not read as applying to the new one. #### Constraint Validation @@ -730,8 +737,8 @@ The status of a constraint for a snapshot is one of: | Status | Description | |---------------|-------------| | `validated` | The constraint was checked and holds for all rows in the snapshot | -| `valid` | The constraint holds for all rows in the snapshot because it was enforced for the commit that produced the snapshot and the parent snapshot's status is `validated` or `valid` | -| `invalid` | The constraint was checked and at least one row in the snapshot violates it | +| `valid` | This write has maintained the constraint for all added rows and the previous snapshot was either `valid` or `validated`. By induction, the constraint holds for all rows in the table. | +| `invalid` | Either the constraint was checked and at least one row in the snapshot violates it, or the snapshot was built on a parent snapshot whose status is `invalid` | | `unvalidated` | Whether the constraint holds for all rows in the snapshot is not known | A snapshot's `constraint-statuses` records, for each status, the IDs of the constraints that have that status for the snapshot: @@ -743,15 +750,12 @@ A snapshot's `constraint-statuses` records, for each status, the IDs of the cons | _optional_ | **`invalid`** | `list` | IDs of constraints that are `invalid` for the snapshot | | _optional_ | **`unvalidated`** | `list` | IDs of constraints that are `unvalidated` for the snapshot | -Each list contains the `constraint-id` of every constraint that has that status for the snapshot. Every constraint that exists when the snapshot is created must be listed in exactly one of the four lists, and a `constraint-id` must not appear in more than one list. A list with no constraints may be omitted. A constraint whose ID is not present in any list did not exist when the snapshot was created, so the snapshot makes no claim about it. - -This is an explicit representation: each constraint's status is recorded independently, so the size of `constraint-statuses` grows with the number of constraints in a table. This keeps the encoding simple; more compact representations may be added in a later version if it becomes a problem. +Each list contains the `constraint-id` of every constraint that has that status for the snapshot. Every constraint that exists when the snapshot is created must be listed in exactly one of the four lists, and a `constraint-id` must not appear in more than one list. A list with no constraints may be omitted. A constraint whose ID is not present in any list did not exist when the snapshot was created, so it is `unvalidated` for the snapshot. Readers must determine the status of a constraint for a snapshot as follows: -1. If the snapshot has no `constraint-statuses`, the snapshot makes no claim about any constraint -2. If the constraint's `constraint-id` is listed in `validated`, `valid`, `invalid`, or `unvalidated`, that is its status -3. Otherwise, the constraint did not exist when the snapshot was created and the snapshot makes no claim about it +1. If the constraint's `constraint-id` is listed in `validated`, `valid`, `invalid`, or `unvalidated`, that is its status +2. Otherwise, the constraint is `unvalidated` for the snapshot Writers must record `constraint-statuses` in every snapshot of a table that has constraints, and must place every constraint that exists when the snapshot is created into exactly one status list, following these rules: @@ -762,17 +766,13 @@ Writers must record `constraint-statuses` in every snapshot of a table that has Enforcing a constraint for a commit is not sufficient to list it as `valid`. When the parent snapshot's status is not `validated` or `valid`, rows added by earlier commits were never checked, so the status is `unvalidated` even though the writer verified the rows that it added. +A commit that does not add rows cannot introduce a violation. A `check` expression is evaluated for each row, and `unique` and `primary-key` require values to be distinct, so removing rows or rewriting files without changing rows preserves all three. Such a commit, for example a delete or a compaction, may record the parent snapshot's `validated` or `valid` status for a constraint without checking rows again. + When a constraint becomes enforced, either by being added with `enforced` set to true or by `enforced` changing from false to true, writers should validate the table and record `validated`. A writer that does not validate records `unvalidated`, and the constraint remains `unvalidated` until a later validation records `validated`. A writer does not have to check every row in a single scan. After checking every row in an ancestor snapshot, a writer may check only the rows added between that ancestor and the current snapshot and record `validated` for the current snapshot. This allows a validation to finish on a table that is written concurrently, without blocking writes or restarting the scan. -A snapshot's `constraint-statuses` must not be modified after the snapshot is created. Recording a different status for a constraint requires a new snapshot. A snapshot that changes only constraint statuses may reuse its parent's manifest list. - -A constraint that a snapshot reports as `valid` may later be found not to hold for that snapshot. A `valid` status depends on every commit in the snapshot's history having correctly enforced the constraint, and Iceberg records the status a writer reports without re-checking the data. So if any of those writers was buggy or non-compliant, the `valid` status can be wrong even though nothing detected it at commit time. A `validated` status, which reflects an actual scan of all rows, does not depend on that chain. Writers should then commit a snapshot that records `invalid` and should expire the snapshots that state that the constraint holds, because queries against those snapshots would otherwise continue to rely on a constraint that does not hold. - -Iceberg does not detect violations on its own. A constraint is only found to be `invalid` when an engine scans the data and checks the constraint, for example during an explicit validation or when validating a newly enforced constraint, and then records `invalid`. Until such a scan runs, a `valid` status that does not actually hold is not detected. - -Only `validated` and `valid` state that a constraint holds. They are distinguished so that readers can tell whether that conclusion was reached by checking data or by relying on writers to enforce the constraint, and so that the most recent `validated` snapshot can be found if a constraint is later found to be `invalid`. +A snapshot's `constraint-statuses` must not be modified after the snapshot is created. Recording a different status for a constraint requires a new snapshot. Writers may commit to a table where a constraint is `invalid`. An enforced constraint requires that a writer not add rows that violate the constraint; it does not require a writer to repair existing violations. @@ -1448,7 +1448,7 @@ Notes: When two commits happen at the same time and are based on the same version, only one commit will succeed. In most cases, the failed commit can be applied to the new current version of table metadata and retried. Updates verify the conditions under which they can be applied to a new version and retry if those conditions are met. -* Append operations have no requirements and can always be applied. +* Append operations have no requirements and can always be applied, unless the table has an enforced `unique` or `primary-key` constraint, which requires verifying the added rows against the new version. See [Constraints](#constraints). * Replace operations must verify that the files that will be deleted are still in the table. Examples of replace operations include format changes (replace an Avro file with a Parquet file) and compactions (several files are replaced with a single file that contains the same rows). * Delete operations must verify that specific files to delete are still in the table. Delete operations based on expressions can always be applied (e.g., where timestamp < X). * Table schema updates and partition spec changes must validate that the schema has not changed between the base version and the current version. @@ -1800,6 +1800,8 @@ Schemas are serialized as a JSON object with the same fields as a struct in the | _optional_ | _required_ |**`schema-id`**|`JSON int`|`0`| | _optional_ | _optional_ |**`identifier-field-ids`**|`JSON list of ints`|`[1, 2]`| +`identifier-field-ids` is not used in v4, where a [`primary-key` constraint](#constraints) expresses the same concept. + Types are serialized according to this table: |Type|JSON representation|Example| From d49816f7736490774e0411345375fb4b9edfa6fa Mon Sep 17 00:00:00 2001 From: Huaxin Gao Date: Wed, 30 Sep 2026 08:59:17 -0700 Subject: [PATCH 13/25] address comments --- format/spec.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/format/spec.md b/format/spec.md index 3c76ab274994..f5959aa2a62b 100644 --- a/format/spec.md +++ b/format/spec.md @@ -771,8 +771,6 @@ A commit that does not add rows cannot introduce a violation. A `check` expressi When a constraint becomes enforced, either by being added with `enforced` set to true or by `enforced` changing from false to true, writers should validate the table and record `validated`. A writer that does not validate records `unvalidated`, and the constraint remains `unvalidated` until a later validation records `validated`. -A writer does not have to check every row in a single scan. After checking every row in an ancestor snapshot, a writer may check only the rows added between that ancestor and the current snapshot and record `validated` for the current snapshot. This allows a validation to finish on a table that is written concurrently, without blocking writes or restarting the scan. - A snapshot's `constraint-statuses` must not be modified after the snapshot is created. Recording a different status for a constraint requires a new snapshot. Writers may commit to a table where a constraint is `invalid`. An enforced constraint requires that a writer not add rows that violate the constraint; it does not require a writer to repair existing violations. From 3703bc0a2d715e7d791153d62644d8d3e62447ec Mon Sep 17 00:00:00 2001 From: Huaxin Gao Date: Thu, 1 Oct 2026 17:44:26 -0700 Subject: [PATCH 14/25] address comments --- format/spec.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/format/spec.md b/format/spec.md index f5959aa2a62b..aff8075b0ee4 100644 --- a/format/spec.md +++ b/format/spec.md @@ -698,7 +698,7 @@ When a commit is retried against a new parent snapshot, the rows that a writer a Whether to trust a constraint that is not enforced is left to engines and is not tracked in table metadata. -A table may have at most one `primary-key` constraint. A `primary-key` constraint replaces [identifier field IDs](#identifier-field-ids), which express the same concept: a set of fields that identifies a row, without a uniqueness guarantee. Identifier field IDs are not used in v4. +A table may have at most one `primary-key` constraint. A `primary-key` constraint replaces [identifier field IDs](#identifier-field-ids), which express the same concept: a set of fields that identifies a row. Iceberg never guaranteed uniqueness for identifier fields and did not track whether it held. A `primary-key` constraint makes both expressible, through `enforced` and `constraint-statuses`. Identifier field IDs are not used in v4. When a table is upgraded to v4, the `identifier-field-ids` of the table's current schema are rewritten as a single `primary-key` constraint that is not enforced. The constraint is assigned a `constraint-id` from `last-constraint-id` in the same way as any other constraint, its `name` is `pk`, and its `timestamp-ms` is the time of the upgrade. A table whose current schema has no identifier fields has no constraints until they are added. Writers must not set `identifier-field-ids` in a schema that is added to a v4 table, and readers must ignore `identifier-field-ids` in a v4 table. @@ -710,7 +710,7 @@ The `expression` of a `check` constraint is serialized as described in the [Iceb A check expression is evaluated for each row over the values of that row. An expression may reference more than one field of the row, such as `start_date <= end_date`. Expressions that depend on more than one row, such as aggregates and window functions, and expressions that depend on another table, such as subqueries, must not be used. -A check expression must produce the same result every time it is evaluated for the same row. A function that depends on anything other than its arguments, such as the current time or a random value, must not be called, because the status recorded for a snapshot describes the table's data and an expression whose result can change on its own would make a recorded status wrong without any write. A [user-defined function](udf-spec.md) records whether it is deterministic. +A check expression must produce the same result every time it is evaluated for the same row. A function that depends on anything other than its arguments, such as the current time or a random value, must not be called, because the status recorded for a snapshot describes the table's data and an expression whose result can change on its own would make a recorded status wrong without any write. A [user-defined function](udf-spec.md) must not be called unless it declares `deterministic` as true. Changing the definition of a function that a check expression calls changes what the constraint requires even though the constraint itself is unchanged. Statuses recorded before the change do not describe the new definition, so a writer that changes such a function should validate the constraint again. From ae49fd995225cf329d266b78ab9ed0a1354cd1fb Mon Sep 17 00:00:00 2001 From: Huaxin Gao Date: Thu, 1 Oct 2026 23:11:10 -0700 Subject: [PATCH 15/25] address comments --- format/spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/format/spec.md b/format/spec.md index aff8075b0ee4..62231b1fe2ad 100644 --- a/format/spec.md +++ b/format/spec.md @@ -731,7 +731,7 @@ Only a constraint's `name` and `enforced` fields may be changed in place, and a #### Constraint Validation -Enforcement and validation are separate properties. Whether writers must verify the rows that they add is a property of a constraint, tracked by `enforced`. Whether a table is known to satisfy a constraint is a property of a table's data, tracked per snapshot by `constraint-statuses`. +Enforcement and constraint status are tracked separately. Whether writers must verify the rows that they add is a property of a constraint, tracked by `enforced`. Whether a table is known to satisfy a constraint is a property of a table's data, tracked per snapshot by `constraint-statuses`. The status of a constraint for a snapshot is one of: From af3f970cbda66c772d499fc0994a85313f782d73 Mon Sep 17 00:00:00 2001 From: Huaxin Gao Date: Mon, 5 Oct 2026 17:11:10 -0700 Subject: [PATCH 16/25] address comments --- format/spec.md | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/format/spec.md b/format/spec.md index 62231b1fe2ad..49616b074b76 100644 --- a/format/spec.md +++ b/format/spec.md @@ -700,7 +700,7 @@ Whether to trust a constraint that is not enforced is left to engines and is not A table may have at most one `primary-key` constraint. A `primary-key` constraint replaces [identifier field IDs](#identifier-field-ids), which express the same concept: a set of fields that identifies a row. Iceberg never guaranteed uniqueness for identifier fields and did not track whether it held. A `primary-key` constraint makes both expressible, through `enforced` and `constraint-statuses`. Identifier field IDs are not used in v4. -When a table is upgraded to v4, the `identifier-field-ids` of the table's current schema are rewritten as a single `primary-key` constraint that is not enforced. The constraint is assigned a `constraint-id` from `last-constraint-id` in the same way as any other constraint, its `name` is `pk`, and its `timestamp-ms` is the time of the upgrade. A table whose current schema has no identifier fields has no constraints until they are added. Writers must not set `identifier-field-ids` in a schema that is added to a v4 table, and readers must ignore `identifier-field-ids` in a v4 table. +When a table is upgraded to v4, the `identifier-field-ids` of the table's current schema are rewritten as a single `primary-key` constraint that is not enforced. The constraint is assigned a `constraint-id` from `last-constraint-id` in the same way as any other constraint, its `name` is `pk`, and its `timestamp-ms` is the time of the upgrade. If any of those fields cannot be used in a `primary-key` constraint, such as a `geometry` or `geography` field, no constraint is created. A table whose current schema has no identifier fields has no constraints until they are added. Writers must not set `identifier-field-ids` in a schema that is added to a v4 table, and readers must ignore `identifier-field-ids` in a v4 table. Constraint IDs are assigned from the table's `last-constraint-id`, which is treated as 0 when it is not present. Writers must assign a new constraint an ID that is higher than the table's current `last-constraint-id` and must update `last-constraint-id` to the highest assigned ID. Constraint IDs must not be reused after the constraint that used an ID is removed, because retained snapshots may still reference the removed ID. Readers must not assume that every `constraint-id` referenced by a snapshot is present in `constraints`. @@ -708,11 +708,11 @@ Constraint IDs are assigned from the table's `last-constraint-id`, which is trea The `expression` of a `check` constraint is serialized as described in the [Iceberg expressions spec](expressions-spec.md) and must use ID references so that it remains bound to the same fields when columns are renamed or reordered. -A check expression is evaluated for each row over the values of that row. An expression may reference more than one field of the row, such as `start_date <= end_date`. Expressions that depend on more than one row, such as aggregates and window functions, and expressions that depend on another table, such as subqueries, must not be used. +A check expression is evaluated for each row over the values of that row. An expression may reference more than one field of the row, such as `start_date <= end_date`. Expressions that depend on more than one row, such as aggregates and window functions, and expressions that depend on another table, such as subqueries, must not be used. An expression must not reference a field within a `list` or a `map`, because such a field has a value for each element rather than one value for the row. A check expression must produce the same result every time it is evaluated for the same row. A function that depends on anything other than its arguments, such as the current time or a random value, must not be called, because the status recorded for a snapshot describes the table's data and an expression whose result can change on its own would make a recorded status wrong without any write. A [user-defined function](udf-spec.md) must not be called unless it declares `deterministic` as true. -Changing the definition of a function that a check expression calls changes what the constraint requires even though the constraint itself is unchanged. Statuses recorded before the change do not describe the new definition, so a writer that changes such a function should validate the constraint again. +The status of a `check` constraint that calls a user-defined function describes the function definitions that were current when the snapshot was written. Iceberg predicates use two-valued logic and null-safe comparisons, as defined in the [expressions spec](expressions-spec.md#boolean-logic). This differs from SQL `CHECK`, where a row satisfies a constraint unless the predicate produces false. @@ -737,9 +737,9 @@ The status of a constraint for a snapshot is one of: | Status | Description | |---------------|-------------| -| `validated` | The constraint was checked and holds for all rows in the snapshot | +| `validated` | The constraint was checked and holds for all rows in the snapshot, or the commit did not add rows and the parent snapshot's status is `validated` | | `valid` | This write has maintained the constraint for all added rows and the previous snapshot was either `valid` or `validated`. By induction, the constraint holds for all rows in the table. | -| `invalid` | Either the constraint was checked and at least one row in the snapshot violates it, or the snapshot was built on a parent snapshot whose status is `invalid` | +| `invalid` | At least one row in the snapshot is known to violate the constraint, either because the constraint was checked or because the parent snapshot's status is `invalid` and the commit did not remove any rows | | `unvalidated` | Whether the constraint holds for all rows in the snapshot is not known | A snapshot's `constraint-statuses` records, for each status, the IDs of the constraints that have that status for the snapshot: @@ -760,8 +760,8 @@ Readers must determine the status of a constraint for a snapshot as follows: Writers must record `constraint-statuses` in every snapshot of a table that has constraints, and must place every constraint that exists when the snapshot is created into exactly one status list, following these rules: -* A constraint must not be listed as `validated` unless it was checked for every row in the snapshot -* A constraint must not be listed as `valid` unless it was enforced for the commit and the parent snapshot's status for the constraint is `validated` or `valid` +* A constraint must not be listed as `validated` unless it was checked for every row in the snapshot, or the commit did not add rows and the parent snapshot's status for the constraint is `validated` +* A constraint must not be listed as `valid` unless the parent snapshot's status for the constraint is `validated` or `valid`, and either it was enforced for the commit or the commit did not add rows * A constraint must not be listed as `invalid` unless a row in the snapshot is known to violate it * `unvalidated` is the status of a constraint that cannot be listed in any other status @@ -771,7 +771,7 @@ A commit that does not add rows cannot introduce a violation. A `check` expressi When a constraint becomes enforced, either by being added with `enforced` set to true or by `enforced` changing from false to true, writers should validate the table and record `validated`. A writer that does not validate records `unvalidated`, and the constraint remains `unvalidated` until a later validation records `validated`. -A snapshot's `constraint-statuses` must not be modified after the snapshot is created. Recording a different status for a constraint requires a new snapshot. +A snapshot's `constraint-statuses` must not be modified after the snapshot is created. Recording a different status for a constraint requires a new snapshot. A snapshot that only records constraint statuses does not add or remove files and uses the `replace` operation. Writers may commit to a table where a constraint is `invalid`. An enforced constraint requires that a writer not add rows that violate the constraint; it does not require a writer to repair existing violations. @@ -1449,10 +1449,11 @@ Notes: When two commits happen at the same time and are based on the same version, only one commit will succeed. In most cases, the failed commit can be applied to the new current version of table metadata and retried. Updates verify the conditions under which they can be applied to a new version and retry if those conditions are met. -* Append operations have no requirements and can always be applied, unless the table has an enforced `unique` or `primary-key` constraint, which requires verifying the added rows against the new version. See [Constraints](#constraints). +* Append operations have no requirements and can always be applied, other than the constraint requirements below. * Replace operations must verify that the files that will be deleted are still in the table. Examples of replace operations include format changes (replace an Avro file with a Parquet file) and compactions (several files are replaced with a single file that contains the same rows). * Delete operations must verify that specific files to delete are still in the table. Delete operations based on expressions can always be applied (e.g., where timestamp < X). * Table schema updates and partition spec changes must validate that the schema has not changed between the base version and the current version. +* For a table with constraints, every operation must also follow the retry requirements in [Constraints](#constraints), including recomputing `constraint-statuses` from the new version. #### File System Tables @@ -2069,7 +2070,7 @@ Constraints are added in v4: * [Constraints](#constraints) must not be added to v3 or earlier tables * Table metadata may contain `constraints` and `last-constraint-id` * Snapshots may contain `constraint-statuses` -* Upgrading a v2 or v3 table to v4 rewrites the table's `identifier-field-ids` as a `primary-key` constraint that is not enforced; a table with no identifier fields has no constraints until they are added +* Upgrading a v2 or v3 table to v4 rewrites the table's `identifier-field-ids` as a `primary-key` constraint that is not enforced, unless a field cannot be used in a `primary-key` constraint; a table with no identifier fields has no constraints until they are added ### Version 3 From 3d6e6fa22d326b0514e438ae5bd7fdb5f92abe2f Mon Sep 17 00:00:00 2001 From: Huaxin Gao Date: Mon, 5 Oct 2026 18:27:05 -0700 Subject: [PATCH 17/25] address comments --- format/spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/format/spec.md b/format/spec.md index 49616b074b76..8893074600c5 100644 --- a/format/spec.md +++ b/format/spec.md @@ -710,7 +710,7 @@ The `expression` of a `check` constraint is serialized as described in the [Iceb A check expression is evaluated for each row over the values of that row. An expression may reference more than one field of the row, such as `start_date <= end_date`. Expressions that depend on more than one row, such as aggregates and window functions, and expressions that depend on another table, such as subqueries, must not be used. An expression must not reference a field within a `list` or a `map`, because such a field has a value for each element rather than one value for the row. -A check expression must produce the same result every time it is evaluated for the same row. A function that depends on anything other than its arguments, such as the current time or a random value, must not be called, because the status recorded for a snapshot describes the table's data and an expression whose result can change on its own would make a recorded status wrong without any write. A [user-defined function](udf-spec.md) must not be called unless it declares `deterministic` as true. +A check expression must produce the same result every time it is evaluated for the same row. A function that depends on anything other than its arguments, such as the current time or a random value, must not be called, because the status recorded for a snapshot describes the table's data and an expression whose result can change on its own would make a recorded status wrong without any write. A [user-defined function](udf-spec.md) must not be called unless its current definition version declares `deterministic` as true. The status of a `check` constraint that calls a user-defined function describes the function definitions that were current when the snapshot was written. From 23479d8916dd60e5d32597ded5e53dd0657d9435 Mon Sep 17 00:00:00 2001 From: Huaxin Gao Date: Sun, 11 Oct 2026 21:38:41 +0100 Subject: [PATCH 18/25] address comments --- format/spec.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/format/spec.md b/format/spec.md index 8893074600c5..d3adb91d95e0 100644 --- a/format/spec.md +++ b/format/spec.md @@ -694,7 +694,7 @@ The fields of a `primary-key` constraint must be `required` and must not be nest When a constraint is `enforced`, writers must only add rows they can prove follow the constraint. A constraint that is not enforced can be written to by any writer, regardless of their ability to prove that the constraint is followed by new rows. -When a commit is retried against a new parent snapshot, the rows that a writer adds must still satisfy the table's enforced constraints. A `check` constraint is evaluated for each row on its own, so a retry cannot change its result. A `unique` or `primary-key` constraint compares rows to each other, so a writer must verify the rows that it adds against the new parent; verifying them against the original parent is not sufficient. A writer must also verify the rows that it adds against any constraint that a concurrent commit added or changed to enforced, and must recompute `constraint-statuses` from the new parent. +When a commit is retried against a new parent snapshot, the rows that a writer adds must still satisfy the table's enforced constraints. A `check` constraint is evaluated for each row on its own, so a retry cannot change its result. A `unique` or `primary-key` constraint compares rows to each other, so a writer must verify that added rows have distinct keys among themselves and do not conflict with rows remaining from the new parent after applying the commit's removals; verifying them against the original parent is not sufficient. A writer must also verify the rows that it adds against any constraint that a concurrent commit added or changed to enforced, and must recompute `constraint-statuses` from the new parent. Whether to trust a constraint that is not enforced is left to engines and is not tracked in table metadata. @@ -708,7 +708,7 @@ Constraint IDs are assigned from the table's `last-constraint-id`, which is trea The `expression` of a `check` constraint is serialized as described in the [Iceberg expressions spec](expressions-spec.md) and must use ID references so that it remains bound to the same fields when columns are renamed or reordered. -A check expression is evaluated for each row over the values of that row. An expression may reference more than one field of the row, such as `start_date <= end_date`. Expressions that depend on more than one row, such as aggregates and window functions, and expressions that depend on another table, such as subqueries, must not be used. An expression must not reference a field within a `list` or a `map`, because such a field has a value for each element rather than one value for the row. +A check expression is evaluated for each row over the values of that row. An expression may reference more than one field of the row, such as `start_date <= end_date`. Expressions that depend on more than one row, such as aggregates over rows and window functions, and expressions that depend on another table, such as subqueries, must not be used. An expression must not reference a field within a `list` or a `map`, because such a field has a value for each element rather than one value for the row. An expression may pass a whole `list` or `map` field to a function, for example to check that a list is not empty. A check expression must produce the same result every time it is evaluated for the same row. A function that depends on anything other than its arguments, such as the current time or a random value, must not be called, because the status recorded for a snapshot describes the table's data and an expression whose result can change on its own would make a recorded status wrong without any write. A [user-defined function](udf-spec.md) must not be called unless its current definition version declares `deterministic` as true. @@ -716,7 +716,7 @@ The status of a `check` constraint that calls a user-defined function describes Iceberg predicates use two-valued logic and null-safe comparisons, as defined in the [expressions spec](expressions-spec.md#boolean-logic). This differs from SQL `CHECK`, where a row satisfies a constraint unless the predicate produces false. -To express SQL `CHECK` semantics for an optional field, the stored expression must make the null case explicit. For example, SQL `CHECK (price >= 0)` for an optional `price` field is stored as the expression for `price >= 0 OR price IS NULL`. This is unnecessary for required fields, which can never be null. +To express SQL `CHECK` semantics for a field that can be null, the stored expression must make the null case explicit. For example, SQL `CHECK (price >= 0)` for an optional `price` field is stored as the expression for `price >= 0 OR price IS NULL`. #### Constraints and Schema Evolution @@ -1103,7 +1103,7 @@ A snapshot consists of the following fields: The snapshot summary's `operation` field is used by some operations, like snapshot expiration, to skip processing certain snapshots. Possible `operation` values are: * `append` -- Only data files were added and no files were removed. -* `replace` -- Data and delete files were added and removed without changing table data; i.e., compaction, changing the data file format, or relocating data files. +* `replace` -- Data and delete files were added and removed without changing table data; i.e., compaction, changing the data file format, or relocating data files. A snapshot that only records [constraint statuses](#constraint-validation) and does not add or remove files also uses `replace`. * `overwrite` -- Data and delete files were added and removed in a logical overwrite operation. * `delete` -- Data files were removed and their contents logically deleted and/or delete files were added to delete rows. From 770ae429afa2db30a659d9935c2ae42ee30e1ab7 Mon Sep 17 00:00:00 2001 From: Huaxin Gao Date: Tue, 13 Oct 2026 08:45:47 +0100 Subject: [PATCH 19/25] address comments --- format/spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/format/spec.md b/format/spec.md index d3adb91d95e0..05464bfa5ef7 100644 --- a/format/spec.md +++ b/format/spec.md @@ -712,7 +712,7 @@ A check expression is evaluated for each row over the values of that row. An exp A check expression must produce the same result every time it is evaluated for the same row. A function that depends on anything other than its arguments, such as the current time or a random value, must not be called, because the status recorded for a snapshot describes the table's data and an expression whose result can change on its own would make a recorded status wrong without any write. A [user-defined function](udf-spec.md) must not be called unless its current definition version declares `deterministic` as true. -The status of a `check` constraint that calls a user-defined function describes the function definitions that were current when the snapshot was written. +The status of a `check` constraint that calls a user-defined function describes the function definitions that were current when the snapshot was written. Changing a function definition can make the status of a constraint that calls it wrong, and engines are responsible for validating the constraint again. Iceberg predicates use two-valued logic and null-safe comparisons, as defined in the [expressions spec](expressions-spec.md#boolean-logic). This differs from SQL `CHECK`, where a row satisfies a constraint unless the predicate produces false. From df8d51f1a90d93b71db485dcd11f92311dc43ec9 Mon Sep 17 00:00:00 2001 From: Huaxin Gao Date: Tue, 13 Oct 2026 23:54:26 +0200 Subject: [PATCH 20/25] address comments --- format/spec.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/format/spec.md b/format/spec.md index 05464bfa5ef7..42201ef9281d 100644 --- a/format/spec.md +++ b/format/spec.md @@ -684,7 +684,7 @@ A constraint consists of the following fields: | _required_ | **`enforced`** | `boolean` | Whether writers must only add rows that satisfy the constraint | | _required_ | **`timestamp-ms`** | `long` | Timestamp in milliseconds from the unix epoch when the constraint was created or last modified. The timestamp is informational and must not be used to determine whether a constraint applies to a snapshot or whether it holds | | _optional_ | **`expression`** | `expression` | The predicate that every row must satisfy, see [Check Constraint Expressions](#check-constraint-expressions). Required for a `check` constraint and must not be set for other types | -| _optional_ | **`field-ids`** | `list` | The list of field IDs that the constraint applies to. Required for a `unique` or `primary-key` constraint and must not be set for a `check` constraint | +| _optional_ | **`field-ids`** | `list` | The set of field IDs that the constraint applies to. Required for a `unique` or `primary-key` constraint and must not be set for a `check` constraint | The fields that define what a constraint requires are embedded directly in the constraint based on its `type`. Each type carries only the metadata that it requires: a `check` constraint has an `expression` and must not declare `field-ids`, and a `unique` or `primary-key` constraint has `field-ids` and must not declare an `expression`. This keeps a single source of truth for the fields that a constraint references. @@ -694,6 +694,8 @@ The fields of a `primary-key` constraint must be `required` and must not be nest When a constraint is `enforced`, writers must only add rows they can prove follow the constraint. A constraint that is not enforced can be written to by any writer, regardless of their ability to prove that the constraint is followed by new rows. +For constraints, a row whose values are changed by a commit counts as a removed row and an added row, whether the engine models the change as deleting and inserting the row or as a modification that preserves the row ID (see [Row Lineage](#row-lineage)). + When a commit is retried against a new parent snapshot, the rows that a writer adds must still satisfy the table's enforced constraints. A `check` constraint is evaluated for each row on its own, so a retry cannot change its result. A `unique` or `primary-key` constraint compares rows to each other, so a writer must verify that added rows have distinct keys among themselves and do not conflict with rows remaining from the new parent after applying the commit's removals; verifying them against the original parent is not sufficient. A writer must also verify the rows that it adds against any constraint that a concurrent commit added or changed to enforced, and must recompute `constraint-statuses` from the new parent. Whether to trust a constraint that is not enforced is left to engines and is not tracked in table metadata. @@ -761,7 +763,7 @@ Readers must determine the status of a constraint for a snapshot as follows: Writers must record `constraint-statuses` in every snapshot of a table that has constraints, and must place every constraint that exists when the snapshot is created into exactly one status list, following these rules: * A constraint must not be listed as `validated` unless it was checked for every row in the snapshot, or the commit did not add rows and the parent snapshot's status for the constraint is `validated` -* A constraint must not be listed as `valid` unless the parent snapshot's status for the constraint is `validated` or `valid`, and either it was enforced for the commit or the commit did not add rows +* A constraint must not be listed as `valid` unless the parent snapshot's status for the constraint is `validated` or `valid`, and either the writer can prove that the rows it added follow the constraint or the commit did not add rows * A constraint must not be listed as `invalid` unless a row in the snapshot is known to violate it * `unvalidated` is the status of a constraint that cannot be listed in any other status @@ -769,7 +771,7 @@ Enforcing a constraint for a commit is not sufficient to list it as `valid`. Whe A commit that does not add rows cannot introduce a violation. A `check` expression is evaluated for each row, and `unique` and `primary-key` require values to be distinct, so removing rows or rewriting files without changing rows preserves all three. Such a commit, for example a delete or a compaction, may record the parent snapshot's `validated` or `valid` status for a constraint without checking rows again. -When a constraint becomes enforced, either by being added with `enforced` set to true or by `enforced` changing from false to true, writers should validate the table and record `validated`. A writer that does not validate records `unvalidated`, and the constraint remains `unvalidated` until a later validation records `validated`. +When a constraint becomes enforced, either by being added with `enforced` set to true or by `enforced` changing from false to true, writers should validate the table and record `validated`. A snapshot's `constraint-statuses` must not be modified after the snapshot is created. Recording a different status for a constraint requires a new snapshot. A snapshot that only records constraint statuses does not add or remove files and uses the `replace` operation. From 833e873ccb897d526998bfc6b0fe9e0173ec12f5 Mon Sep 17 00:00:00 2001 From: Huaxin Gao Date: Thu, 15 Oct 2026 21:04:01 +0200 Subject: [PATCH 21/25] address comments --- format/spec.md | 66 ++++++++++++++++++++------------------------------ 1 file changed, 26 insertions(+), 40 deletions(-) diff --git a/format/spec.md b/format/spec.md index 42201ef9281d..edcb3282e360 100644 --- a/format/spec.md +++ b/format/spec.md @@ -437,7 +437,7 @@ Two rows are the "same"---that is, the rows represent the same entity---if the i Identifier fields may be nested in structs but cannot be nested within maps or lists. Float, double, and optional fields cannot be used as identifier fields and a nested field cannot be used as an identifier field if it is nested in an optional struct, to avoid null values in identifiers. -Identifier field IDs are not used in v4, where a [`primary-key` constraint](#constraints) expresses the same concept. +Identifier field IDs must not be used in v4, where a [`primary-key` constraint](#constraints) expresses the same concept. #### Reserved Field IDs @@ -662,7 +662,7 @@ A data or delete file is associated with a sort order by the sort order's id wit Constraints are added in v4 and are not supported in v3 or earlier. -A **constraint** declares a condition that a table's rows are expected to satisfy. Its definition is stored in table metadata. Each snapshot contains the writer-reported status of each constraint that existed when the snapshot was created; see [Constraint Validation](#constraint-validation). +A **constraint** declares a condition that a table's live rows, after applying deletes, are expected to satisfy. Its definition is stored in table metadata. Each snapshot contains the writer-reported status of each constraint that existed when the snapshot was created; see [Constraint Validation](#constraint-validation). Three constraint types are defined: @@ -684,52 +684,45 @@ A constraint consists of the following fields: | _required_ | **`enforced`** | `boolean` | Whether writers must only add rows that satisfy the constraint | | _required_ | **`timestamp-ms`** | `long` | Timestamp in milliseconds from the unix epoch when the constraint was created or last modified. The timestamp is informational and must not be used to determine whether a constraint applies to a snapshot or whether it holds | | _optional_ | **`expression`** | `expression` | The predicate that every row must satisfy, see [Check Constraint Expressions](#check-constraint-expressions). Required for a `check` constraint and must not be set for other types | -| _optional_ | **`field-ids`** | `list` | The set of field IDs that the constraint applies to. Required for a `unique` or `primary-key` constraint and must not be set for a `check` constraint | +| _optional_ | **`field-ids`** | `list` | The set of field IDs that the constraint applies to. Must contain at least one field ID and no duplicates. Required for a `unique` or `primary-key` constraint and must not be set for a `check` constraint | -The fields that define what a constraint requires are embedded directly in the constraint based on its `type`. Each type carries only the metadata that it requires: a `check` constraint has an `expression` and must not declare `field-ids`, and a `unique` or `primary-key` constraint has `field-ids` and must not declare an `expression`. This keeps a single source of truth for the fields that a constraint references. +The `field-ids` of a `unique` or `primary-key` constraint must only reference primitive fields and must not reference fields within a `list` or a `map`. Fields of type `float` or `double` must not be used. Fields of type `geometry` or `geography` must not be used because Iceberg does not define equality for them. Fields of type `unknown` must not be used because their values are always null. -The `field-ids` of a `unique` or `primary-key` constraint must reference primitive fields and must not reference fields within a `list` or a `map`. Fields of type `float`, `double`, `geometry`, or `geography` must not be used because Iceberg does not define equality for them. Fields of type `unknown` must not be used because their values are always null. +The fields of a `primary-key` constraint must be `required` and must not be nested in an optional struct, so that the schema rules out null keys even when the constraint is not enforced. The fields of a `unique` constraint may be optional or nested in optional structs, because a row in which any field is null is not compared. -The fields of a `primary-key` constraint must be `required` and must not be nested in an optional struct, so that the schema rules out null keys even when the constraint is not enforced. These are the same restrictions that apply to [identifier fields](#identifier-field-ids). The fields of a `unique` constraint may be optional or nested in optional structs, because a row in which any field is null is not compared. +When a constraint is `enforced`, writers must ensure that written data conform to the constraint. When a constraint is not enforced, writers are not required to ensure that written data conform to the constraint. -When a constraint is `enforced`, writers must only add rows they can prove follow the constraint. A constraint that is not enforced can be written to by any writer, regardless of their ability to prove that the constraint is followed by new rows. - -For constraints, a row whose values are changed by a commit counts as a removed row and an added row, whether the engine models the change as deleting and inserting the row or as a modification that preserves the row ID (see [Row Lineage](#row-lineage)). +For constraints, a row whose values changed in a commit counts as a removed row and an added row, whether the engine models the change as deleting and inserting the row or as a modification that preserves the row ID (see [Row Lineage](#row-lineage)). When a commit is retried against a new parent snapshot, the rows that a writer adds must still satisfy the table's enforced constraints. A `check` constraint is evaluated for each row on its own, so a retry cannot change its result. A `unique` or `primary-key` constraint compares rows to each other, so a writer must verify that added rows have distinct keys among themselves and do not conflict with rows remaining from the new parent after applying the commit's removals; verifying them against the original parent is not sufficient. A writer must also verify the rows that it adds against any constraint that a concurrent commit added or changed to enforced, and must recompute `constraint-statuses` from the new parent. -Whether to trust a constraint that is not enforced is left to engines and is not tracked in table metadata. +If a constraint is not enforced, readers can decide whether to trust the constraint. -A table may have at most one `primary-key` constraint. A `primary-key` constraint replaces [identifier field IDs](#identifier-field-ids), which express the same concept: a set of fields that identifies a row. Iceberg never guaranteed uniqueness for identifier fields and did not track whether it held. A `primary-key` constraint makes both expressible, through `enforced` and `constraint-statuses`. Identifier field IDs are not used in v4. +A table may have at most one `primary-key` constraint. A `primary-key` constraint replaces [identifier field IDs](#identifier-field-ids), which express the same concept: a set of fields that identifies a row. Iceberg never guaranteed uniqueness for identifier fields and did not track whether it held. A `primary-key` constraint makes both expressible, through `enforced` and `constraint-statuses`. -When a table is upgraded to v4, the `identifier-field-ids` of the table's current schema are rewritten as a single `primary-key` constraint that is not enforced. The constraint is assigned a `constraint-id` from `last-constraint-id` in the same way as any other constraint, its `name` is `pk`, and its `timestamp-ms` is the time of the upgrade. If any of those fields cannot be used in a `primary-key` constraint, such as a `geometry` or `geography` field, no constraint is created. A table whose current schema has no identifier fields has no constraints until they are added. Writers must not set `identifier-field-ids` in a schema that is added to a v4 table, and readers must ignore `identifier-field-ids` in a v4 table. +When a table is upgraded to v4, the `identifier-field-ids` of the table's current schema are rewritten as a single `primary-key` constraint that is not enforced. The constraint is assigned a `constraint-id` from `last-constraint-id` in the same way as any other constraint, its `name` is `pk`, and its `timestamp-ms` is the time of the upgrade. If any of those fields cannot be used in a `primary-key` constraint, such as a `geometry` or `geography` field, no constraint is created and the upgrade continues without it. Writers must not set `identifier-field-ids` in a schema that is added to a v4 table, and readers must ignore `identifier-field-ids` in a v4 table. Constraint IDs are assigned from the table's `last-constraint-id`, which is treated as 0 when it is not present. Writers must assign a new constraint an ID that is higher than the table's current `last-constraint-id` and must update `last-constraint-id` to the highest assigned ID. Constraint IDs must not be reused after the constraint that used an ID is removed, because retained snapshots may still reference the removed ID. Readers must not assume that every `constraint-id` referenced by a snapshot is present in `constraints`. #### Check Constraint Expressions -The `expression` of a `check` constraint is serialized as described in the [Iceberg expressions spec](expressions-spec.md) and must use ID references so that it remains bound to the same fields when columns are renamed or reordered. +The `expression` of a `check` constraint is serialized as described in the [Iceberg expressions spec](expressions-spec.md) and must use ID references so that it remains bound to the same fields when columns are renamed. A check expression is evaluated for each row over the values of that row. An expression may reference more than one field of the row, such as `start_date <= end_date`. Expressions that depend on more than one row, such as aggregates over rows and window functions, and expressions that depend on another table, such as subqueries, must not be used. An expression must not reference a field within a `list` or a `map`, because such a field has a value for each element rather than one value for the row. An expression may pass a whole `list` or `map` field to a function, for example to check that a list is not empty. -A check expression must produce the same result every time it is evaluated for the same row. A function that depends on anything other than its arguments, such as the current time or a random value, must not be called, because the status recorded for a snapshot describes the table's data and an expression whose result can change on its own would make a recorded status wrong without any write. A [user-defined function](udf-spec.md) must not be called unless its current definition version declares `deterministic` as true. - -The status of a `check` constraint that calls a user-defined function describes the function definitions that were current when the snapshot was written. Changing a function definition can make the status of a constraint that calls it wrong, and engines are responsible for validating the constraint again. - -Iceberg predicates use two-valued logic and null-safe comparisons, as defined in the [expressions spec](expressions-spec.md#boolean-logic). This differs from SQL `CHECK`, where a row satisfies a constraint unless the predicate produces false. +A check expression must produce the same result every time it is evaluated for the same row. A function that depends on anything other than its arguments, such as the current time or a random value, must not be called. A [user-defined function](udf-spec.md) must not be called unless its current definition version declares `deterministic` as true. -To express SQL `CHECK` semantics for a field that can be null, the stored expression must make the null case explicit. For example, SQL `CHECK (price >= 0)` for an optional `price` field is stored as the expression for `price >= 0 OR price IS NULL`. +The status of a `check` constraint that calls a user-defined function describes the function definitions that were current when the snapshot was written. Changing a function definition can make the status of a constraint that calls it wrong, and engines are responsible for validating the constraint again. Because `constraint-statuses` does not record which function definitions were used, a writer must not record `valid` for such a constraint or keep the parent snapshot's status for it; the constraint is `validated` only if it was checked for every row in the snapshot. #### Constraints and Schema Evolution A constraint references fields by ID, so schema changes interact with constraints as follows. The referenced fields of a `check` constraint are the field IDs in its `expression`; the referenced fields of a `unique` or `primary-key` constraint are its `field-ids`. -* Renaming or reordering a referenced field is allowed; the constraint continues to apply to the same fields. -* Every field referenced by a constraint must exist in the table's current schema. A schema change that removes a referenced field must be rejected unless the constraint is removed in the same change. -* The type of a field referenced by a `check` constraint must not be changed, even for type promotions that are otherwise allowed, because the constraint's `expression` contains literals that are bound to the field's type. The type of a field referenced by a `unique` or `primary-key` constraint may be changed by a type promotion, which preserves values and therefore preserves distinctness. -* A field referenced by a `primary-key` constraint must not be made optional, and a struct that contains such a field must not be made optional, because a `primary-key` requires its fields to be non-null. +* Renaming or reordering a constraint-referenced field is allowed; the constraint continues to apply to the same fields. +* Every field referenced by a constraint must exist in the table's current schema. A schema change that removes a constraint-referenced field must be rejected unless the constraint is removed in the same change. +* A field referenced by a `check` constraint must not change type or apply type promotion, because the constraint's `expression` contains literals that are bound to the field's type. A field referenced by a `unique` or `primary-key` constraint may apply type promotion, because type promotion preserves distinctness. -Only a constraint's `name` and `enforced` fields may be changed in place, and a writer that changes either must update `timestamp-ms`. Changing the `expression` of a `check` constraint or the `field-ids` of a `unique` or `primary-key` constraint changes what the constraint requires, so it must be done by removing the constraint and adding a new one with a new `constraint-id`, so that statuses recorded for the old definition are not read as applying to the new one. +Only `name` and `enforced` fields in a constraint definition can be updated, and `timestamp-ms` must be updated when the constraint is updated. The `expression` and `field-ids` fields must not change, as it would change the constraint requirement. Such updates must be done by removing the constraint and adding a new one with a new `constraint-id`. A new `constraint-id` ensures that statuses recorded for the old definition are not applied to the new one. #### Constraint Validation @@ -740,7 +733,7 @@ The status of a constraint for a snapshot is one of: | Status | Description | |---------------|-------------| | `validated` | The constraint was checked and holds for all rows in the snapshot, or the commit did not add rows and the parent snapshot's status is `validated` | -| `valid` | This write has maintained the constraint for all added rows and the previous snapshot was either `valid` or `validated`. By induction, the constraint holds for all rows in the table. | +| `valid` | The parent snapshot's status is `validated` or `valid`, and either the writer can prove that the rows it added follow the constraint or the commit did not add rows. By induction, the constraint holds for all rows in the snapshot. | | `invalid` | At least one row in the snapshot is known to violate the constraint, either because the constraint was checked or because the parent snapshot's status is `invalid` and the commit did not remove any rows | | `unvalidated` | Whether the constraint holds for all rows in the snapshot is not known | @@ -760,20 +753,13 @@ Readers must determine the status of a constraint for a snapshot as follows: 1. If the constraint's `constraint-id` is listed in `validated`, `valid`, `invalid`, or `unvalidated`, that is its status 2. Otherwise, the constraint is `unvalidated` for the snapshot -Writers must record `constraint-statuses` in every snapshot of a table that has constraints, and must place every constraint that exists when the snapshot is created into exactly one status list, following these rules: - -* A constraint must not be listed as `validated` unless it was checked for every row in the snapshot, or the commit did not add rows and the parent snapshot's status for the constraint is `validated` -* A constraint must not be listed as `valid` unless the parent snapshot's status for the constraint is `validated` or `valid`, and either the writer can prove that the rows it added follow the constraint or the commit did not add rows -* A constraint must not be listed as `invalid` unless a row in the snapshot is known to violate it -* `unvalidated` is the status of a constraint that cannot be listed in any other status - -Enforcing a constraint for a commit is not sufficient to list it as `valid`. When the parent snapshot's status is not `validated` or `valid`, rows added by earlier commits were never checked, so the status is `unvalidated` even though the writer verified the rows that it added. +Writers must record `constraint-statuses` in every snapshot of a table that has constraints, and must place every constraint that exists when the snapshot is created into exactly one status list. A constraint must not be listed with a status unless the status's description above holds for the snapshot; `unvalidated` is the status of a constraint that cannot be listed with any other status. -A commit that does not add rows cannot introduce a violation. A `check` expression is evaluated for each row, and `unique` and `primary-key` require values to be distinct, so removing rows or rewriting files without changing rows preserves all three. Such a commit, for example a delete or a compaction, may record the parent snapshot's `validated` or `valid` status for a constraint without checking rows again. +A commit that does not add rows cannot introduce a violation. Such a commit, for example a delete or a compaction operation, may carry over the parent snapshot's `validated` or `valid` status for a constraint without checking rows again. When a constraint becomes enforced, either by being added with `enforced` set to true or by `enforced` changing from false to true, writers should validate the table and record `validated`. -A snapshot's `constraint-statuses` must not be modified after the snapshot is created. Recording a different status for a constraint requires a new snapshot. A snapshot that only records constraint statuses does not add or remove files and uses the `replace` operation. +A snapshot's `constraint-statuses` must not be modified after the snapshot is created. A `constraint-statuses` update requires a new snapshot. A snapshot that only changes constraint statuses must use the `replace` operation, as it does not add or remove files. Writers may commit to a table where a constraint is `invalid`. An enforced constraint requires that a writer not add rows that violate the constraint; it does not require a writer to repair existing violations. @@ -1100,7 +1086,7 @@ A snapshot consists of the following fields: | _required_ | **`first-row-id`** | The first `_row_id` assigned to the first row in the first data file in the first manifest, see [Row Lineage](#row-lineage) | | _required_ | **`added-rows`** | The upper bound of the number of rows with assigned row IDs, see [Row Lineage](#row-lineage) | | _optional_ | **`key-id`** | ID of the encryption key that encrypts the manifest list key metadata | - | _optional_ | **`constraint-statuses`** | A struct (optional) that records the status of the table's constraints for the snapshot, see [Constraint Validation](#constraint-validation) | + | _optional_ | **`constraint-statuses`** | A struct that records the status of the table's constraints for the snapshot, see [Constraint Validation](#constraint-validation) | The snapshot summary's `operation` field is used by some operations, like snapshot expiration, to skip processing certain snapshots. Possible `operation` values are: @@ -1334,7 +1320,7 @@ Table metadata consists of the following fields: | _required_ | **`next-row-id`** | A `long` higher than all assigned row IDs; the next snapshot's `first-row-id`. See [Row Lineage](#row-lineage). | | _optional_ | **`encryption-keys`** | A list (optional) of [encryption keys](#encryption-keys) used for table encryption. | | _optional_ | **`last-constraint-id`** | An integer; the highest assigned constraint ID for the table. This is used to ensure constraints are always assigned an unused ID. See [Constraints](#constraints). | - | _optional_ | **`constraints`** | A list (optional) of [constraints](#constraints) for the table. | + | _optional_ | **`constraints`** | A list of [constraints](#constraints) for the table. | For serialization details, see Appendix C. @@ -1806,7 +1792,7 @@ Schemas are serialized as a JSON object with the same fields as a struct in the | _optional_ | _required_ |**`schema-id`**|`JSON int`|`0`| | _optional_ | _optional_ |**`identifier-field-ids`**|`JSON list of ints`|`[1, 2]`| -`identifier-field-ids` is not used in v4, where a [`primary-key` constraint](#constraints) expresses the same concept. +`identifier-field-ids` must not be used in v4, where a [`primary-key` constraint](#constraints) expresses the same concept. Types are serialized according to this table: @@ -1935,7 +1921,7 @@ A metadata JSON file may be compressed with [GZIP](https://datatracker.ietf.org/ |**`last-partition-id`**|`JSON int`|`1000`| |**`properties`**|`JSON object: {`
  `"": "",`
  `...`
`}`|`{`
  `"write.format.default": "avro",`
  `"commit.retry.num-retries": "4"`
`}`| |**`current-snapshot-id`**|`JSON long`|`3051729675574597004`| -|**`snapshots`**|`JSON list of objects: [ {`
  `"snapshot-id": ,`
  `"timestamp-ms": ,`
  `"summary": {`
    `"operation": ,`
    `... },`
  `"manifest-list": "",`
  `"schema-id": "",`
  `"constraint-statuses": {`
    `"validated": [ ... ],`
    `... }`
  `},`
  `...`
`]`|`[ {`
  `"snapshot-id": 3051729675574597004,`
  `"timestamp-ms": 1515100955770,`
  `"summary": {`
    `"operation": "append"`
  `},`
  `"manifest-list": "s3://b/wh/.../s1.avro"`
  `"schema-id": 0,`
  `"constraint-statuses": {`
    `"validated": [1],`
    `"unvalidated": [2]`
  `}`
`} ]`| +|**`snapshots`**|`JSON list of objects: [ {`
  `"snapshot-id": ,`
  `"timestamp-ms": ,`
  `"summary": {`
    `"operation": ,`
    `... },`
  `"manifest-list": "",`
  `"schema-id": "",`
  `"constraint-statuses": {`
    `"validated": [ ... ],`
    `... }`
  `},`
  `...`
`]`|`[ {`
  `"snapshot-id": 3051729675574597004,`
  `"timestamp-ms": 1515100955770,`
  `"summary": {`
    `"operation": "append"`
  `},`
  `"manifest-list": "s3://b/wh/.../s1.avro",`
  `"schema-id": 0,`
  `"constraint-statuses": {`
    `"validated": [1],`
    `"unvalidated": [2]`
  `}`
`} ]`| |**`snapshot-log`**|`JSON list of objects: [`
  `{`
  `"snapshot-id": ,`
  `"timestamp-ms":`
  `},`
  `...`
`]`|`[ {`
  `"snapshot-id": 30517296...,`
  `"timestamp-ms": 1515100...`
`} ]`| |**`metadata-log`**|`JSON list of objects: [`
  `{`
  `"metadata-file": ,`
  `"timestamp-ms":`
  `},`
  `...`
`]`|`[ {`
  `"metadata-file": "s3://bucket/.../v1.json",`
  `"timestamp-ms": 1515100...`
`} ]` | |**`sort-orders`**|`JSON sort orders (list of sort field object)`|`See above`| @@ -2072,7 +2058,7 @@ Constraints are added in v4: * [Constraints](#constraints) must not be added to v3 or earlier tables * Table metadata may contain `constraints` and `last-constraint-id` * Snapshots may contain `constraint-statuses` -* Upgrading a v2 or v3 table to v4 rewrites the table's `identifier-field-ids` as a `primary-key` constraint that is not enforced, unless a field cannot be used in a `primary-key` constraint; a table with no identifier fields has no constraints until they are added +* Upgrading a v2 or v3 table to v4 rewrites the table's `identifier-field-ids` as a `primary-key` constraint that is not enforced, unless a field cannot be used in a `primary-key` constraint ### Version 3 From dc215724c2272739469536b164a56d346c7216c0 Mon Sep 17 00:00:00 2001 From: Huaxin Gao Date: Thu, 15 Oct 2026 21:05:55 +0200 Subject: [PATCH 22/25] move constraint-statuses into v4 snapshot table Co-authored-by: Cursor --- format/spec.md | 16 +--------------- 1 file changed, 1 insertion(+), 15 deletions(-) diff --git a/format/spec.md b/format/spec.md index 0ef3441fa3a3..a7ee0009248c 100644 --- a/format/spec.md +++ b/format/spec.md @@ -1206,21 +1206,6 @@ A snapshot consists of the following fields: | | | _required_ | **`first-row-id`** | The first `_row_id` assigned to the first row in the first data file in the first manifest, see [Row Lineage](#row-lineage) | | | | _required_ | **`added-rows`** | The upper bound of the number of rows with assigned row IDs, see [Row Lineage](#row-lineage) | | | | _optional_ | **`key-id`** | ID of the encryption key that encrypts the manifest list key metadata | -=== "v4" - | v4 | Field | Description | - |------------|------------------------------|-------------| - | _required_ | **`snapshot-id`** | A unique long ID | - | _optional_ | **`parent-snapshot-id`** | The snapshot ID of the snapshot's parent. Omitted for any snapshot with no parent | - | _required_ | **`sequence-number`** | A monotonically increasing long that tracks the order of changes to a table | - | _required_ | **`timestamp-ms`** | A timestamp when the snapshot was created, used for garbage collection and table inspection | - | _required_ | **`manifest-list`** | The location of a manifest list for this snapshot that tracks manifest files with additional metadata | - | _required_ | **`summary`** | A string map that summarizes the snapshot changes, including `operation` as a _required_ field (see below) | - | _optional_ | **`schema-id`** | ID of the table's current schema when the snapshot was created | - | _required_ | **`first-row-id`** | The first `_row_id` assigned to the first row in the first data file in the first manifest, see [Row Lineage](#row-lineage) | - | _required_ | **`added-rows`** | The upper bound of the number of rows with assigned row IDs, see [Row Lineage](#row-lineage) | - | _optional_ | **`key-id`** | ID of the encryption key that encrypts the manifest list key metadata | - | _optional_ | **`constraint-statuses`** | A struct that records the status of the table's constraints for the snapshot, see [Constraint Validation](#constraint-validation) | - === "v4" | v4 | Field | Description | | ---------- |------------------------------|-------------| @@ -1234,6 +1219,7 @@ A snapshot consists of the following fields: | _required_ | **`first-row-id`** | The first `_row_id` assigned to the first row in the first data file in the first manifest, see [Row Lineage](#row-lineage) | | _required_ | **`added-rows`** | The upper bound of the number of rows with assigned row IDs, see [Row Lineage](#row-lineage) | | _optional_ | **`key-id`** | ID of the encryption key that encrypts the root manifest key metadata | + | _optional_ | **`constraint-statuses`** | A struct that records the status of the table's constraints for the snapshot, see [Constraint Validation](#constraint-validation) | When committed while the table is at version 4 or higher, a snapshot's `timestamp-ms` must be strictly greater than its parent snapshot's `timestamp-ms` on the same branch. Snapshots with no parent, and snapshots that were committed before a table was upgraded to v4 or higher, do not have this requirement. From 71a1a3d91a04f296d83e3555b88509d1238b1aef Mon Sep 17 00:00:00 2001 From: Huaxin Gao Date: Fri, 16 Oct 2026 09:16:14 +0200 Subject: [PATCH 23/25] address comments --- format/spec.md | 60 +++++++++++++++++++++++++++----------------------- 1 file changed, 32 insertions(+), 28 deletions(-) diff --git a/format/spec.md b/format/spec.md index a7ee0009248c..1f944f0daad8 100644 --- a/format/spec.md +++ b/format/spec.md @@ -443,7 +443,7 @@ Two rows are the "same"---that is, the rows represent the same entity---if the i Identifier fields may be nested in structs but cannot be nested within maps or lists. Float, double, and optional fields cannot be used as identifier fields and a nested field cannot be used as an identifier field if it is nested in an optional struct, to avoid null values in identifiers. -Identifier field IDs must not be used in v4, where a [`primary-key` constraint](#constraints) expresses the same concept. +Identifier field IDs are replaced by a [`primary-key` constraint](#constraints) in v4 and must not be set in a v4 table schema. #### Reserved Field IDs @@ -668,7 +668,7 @@ A data or delete file is associated with a sort order by the sort order's id wit Constraints are added in v4 and are not supported in v3 or earlier. -A **constraint** declares a condition that a table's live rows, after applying deletes, are expected to satisfy. Its definition is stored in table metadata. Each snapshot contains the writer-reported status of each constraint that existed when the snapshot was created; see [Constraint Validation](#constraint-validation). +A **constraint** declares a condition that a table's live rows are expected to satisfy. Its definition is stored in table metadata. Each snapshot contains the writer-reported status of each constraint that existed when the snapshot was created; see [Constraint Validation](#constraint-validation). Three constraint types are defined: @@ -687,38 +687,40 @@ A constraint consists of the following fields: | _required_ | **`constraint-id`** | `int` | ID of the constraint; unique within the table | | _required_ | **`type`** | `string` | The constraint type: `check`, `unique`, or `primary-key` | | _required_ | **`name`** | `string` | A name for the constraint that is unique within the table. Names are for human consumption and must not be used to identify a constraint in metadata | -| _required_ | **`enforced`** | `boolean` | Whether writers must only add rows that satisfy the constraint | +| _required_ | **`enforced`** | `boolean` | Whether writers must only add or modify rows so that they satisfy the constraint | | _required_ | **`timestamp-ms`** | `long` | Timestamp in milliseconds from the unix epoch when the constraint was created or last modified. The timestamp is informational and must not be used to determine whether a constraint applies to a snapshot or whether it holds | | _optional_ | **`expression`** | `expression` | The predicate that every row must satisfy, see [Check Constraint Expressions](#check-constraint-expressions). Required for a `check` constraint and must not be set for other types | | _optional_ | **`field-ids`** | `list` | The set of field IDs that the constraint applies to. Must contain at least one field ID and no duplicates. Required for a `unique` or `primary-key` constraint and must not be set for a `check` constraint | -The `field-ids` of a `unique` or `primary-key` constraint must only reference primitive fields and must not reference fields within a `list` or a `map`. Fields of type `float` or `double` must not be used. Fields of type `geometry` or `geography` must not be used because Iceberg does not define equality for them. Fields of type `unknown` must not be used because their values are always null. +The `field-ids` of a `unique` or `primary-key` constraint must only reference primitive fields and must not reference fields within a `list` or a `map`. Fields of type `float` or `double` must not be used. Fields of type `geometry` or `geography` must not be used because Iceberg does not define equality for them. Fields of type `unknown` must not be used because their values are always null. Fields of type `file` and their sub-fields must not be used. The fields of a `primary-key` constraint must be `required` and must not be nested in an optional struct, so that the schema rules out null keys even when the constraint is not enforced. The fields of a `unique` constraint may be optional or nested in optional structs, because a row in which any field is null is not compared. When a constraint is `enforced`, writers must ensure that written data conform to the constraint. When a constraint is not enforced, writers are not required to ensure that written data conform to the constraint. -For constraints, a row whose values changed in a commit counts as a removed row and an added row, whether the engine models the change as deleting and inserting the row or as a modification that preserves the row ID (see [Row Lineage](#row-lineage)). +When a commit is retried against a new parent snapshot, every enforced constraint, including any that a concurrent commit added or changed to enforced, must hold for the rows that the commit adds or modifies in the snapshot that results from applying the commit to the new parent. `constraint-statuses` must be recomputed from the new parent. -When a commit is retried against a new parent snapshot, the rows that a writer adds must still satisfy the table's enforced constraints. A `check` constraint is evaluated for each row on its own, so a retry cannot change its result. A `unique` or `primary-key` constraint compares rows to each other, so a writer must verify that added rows have distinct keys among themselves and do not conflict with rows remaining from the new parent after applying the commit's removals; verifying them against the original parent is not sufficient. A writer must also verify the rows that it adds against any constraint that a concurrent commit added or changed to enforced, and must recompute `constraint-statuses` from the new parent. +Readers may trust constraints even if not marked as enforced. -If a constraint is not enforced, readers can decide whether to trust the constraint. +A table may have at most one `primary-key` constraint. A `primary-key` constraint with `enforced=false` replaces [identifier field IDs](#identifier-field-ids), which express the same concept: a set of fields that identifies a row, which is not enforced. -A table may have at most one `primary-key` constraint. A `primary-key` constraint replaces [identifier field IDs](#identifier-field-ids), which express the same concept: a set of fields that identifies a row. Iceberg never guaranteed uniqueness for identifier fields and did not track whether it held. A `primary-key` constraint makes both expressible, through `enforced` and `constraint-statuses`. - -When a table is upgraded to v4, the `identifier-field-ids` of the table's current schema are rewritten as a single `primary-key` constraint that is not enforced. The constraint is assigned a `constraint-id` from `last-constraint-id` in the same way as any other constraint, its `name` is `pk`, and its `timestamp-ms` is the time of the upgrade. If any of those fields cannot be used in a `primary-key` constraint, such as a `geometry` or `geography` field, no constraint is created and the upgrade continues without it. Writers must not set `identifier-field-ids` in a schema that is added to a v4 table, and readers must ignore `identifier-field-ids` in a v4 table. +When a table is upgraded to v4, the `identifier-field-ids` of the table's current schema are rewritten as a single `primary-key` constraint that is not enforced. The constraint is assigned a `constraint-id` from `last-constraint-id` in the same way as any other constraint, its `name` is `pk`, and its `timestamp-ms` is the time of the upgrade. If any of those fields cannot be used in a `primary-key` constraint, such as a `geometry` or `geography` field, no constraint is created and the upgrade continues without it. Writers must not set `identifier-field-ids` in a v4 table schema, and readers must ignore `identifier-field-ids` in a v4 table. Constraint IDs are assigned from the table's `last-constraint-id`, which is treated as 0 when it is not present. Writers must assign a new constraint an ID that is higher than the table's current `last-constraint-id` and must update `last-constraint-id` to the highest assigned ID. Constraint IDs must not be reused after the constraint that used an ID is removed, because retained snapshots may still reference the removed ID. Readers must not assume that every `constraint-id` referenced by a snapshot is present in `constraints`. #### Check Constraint Expressions -The `expression` of a `check` constraint is serialized as described in the [Iceberg expressions spec](expressions-spec.md) and must use ID references so that it remains bound to the same fields when columns are renamed. +The `expression` of a `check` constraint is a [predicate](expressions-spec.md#predicates), not a value expression, and is serialized as described in the [Iceberg expressions spec](expressions-spec.md). It must use ID references so that it remains bound to the same fields when columns are renamed. + +A check expression is evaluated for each row over the values of that row. An expression may reference more than one field of the row, such as `start_date <= end_date`. A check expression: -A check expression is evaluated for each row over the values of that row. An expression may reference more than one field of the row, such as `start_date <= end_date`. Expressions that depend on more than one row, such as aggregates over rows and window functions, and expressions that depend on another table, such as subqueries, must not be used. An expression must not reference a field within a `list` or a `map`, because such a field has a value for each element rather than one value for the row. An expression may pass a whole `list` or `map` field to a function, for example to check that a list is not empty. +* Must not depend on more than one row, such as aggregates over rows and window functions +* Must not depend on another table, such as subqueries +* Must not reference a field within a `list` or a `map`, because such a field has a value for each element rather than one value for the row; it may pass a whole `list` or `map` field to a function, for example to check that a list is not empty -A check expression must produce the same result every time it is evaluated for the same row. A function that depends on anything other than its arguments, such as the current time or a random value, must not be called. A [user-defined function](udf-spec.md) must not be called unless its current definition version declares `deterministic` as true. +A check expression must produce the same result every time it is evaluated for the same row. A function that depends on anything other than its arguments, such as the current time or a random value, must not be called. A [user-defined function](udf-spec.md) must not be called unless its current definition version declares `deterministic` as true. A user-defined function is a function loaded from a catalog as described in the UDF spec. Functions in the reserved `sql_functions` and `iceberg_functions` sets and engine built-in functions, as described in the [expressions spec](expressions-spec.md#apply-function), are not user-defined functions. -The status of a `check` constraint that calls a user-defined function describes the function definitions that were current when the snapshot was written. Changing a function definition can make the status of a constraint that calls it wrong, and engines are responsible for validating the constraint again. Because `constraint-statuses` does not record which function definitions were used, a writer must not record `valid` for such a constraint or keep the parent snapshot's status for it; the constraint is `validated` only if it was checked for every row in the snapshot. +A function definition can change without a table commit, and `constraint-statuses` does not record which function definitions were used, so a status for a `check` constraint that calls a user-defined function could become wrong without any write. Such a constraint must always be listed as `unvalidated`. When it is enforced, writers still evaluate it with the current function definitions for the rows that they add or modify. #### Constraints and Schema Evolution @@ -726,21 +728,21 @@ A constraint references fields by ID, so schema changes interact with constraint * Renaming or reordering a constraint-referenced field is allowed; the constraint continues to apply to the same fields. * Every field referenced by a constraint must exist in the table's current schema. A schema change that removes a constraint-referenced field must be rejected unless the constraint is removed in the same change. -* A field referenced by a `check` constraint must not change type or apply type promotion, because the constraint's `expression` contains literals that are bound to the field's type. A field referenced by a `unique` or `primary-key` constraint may apply type promotion, because type promotion preserves distinctness. +* A field referenced by a `check` constraint must not change type or apply type promotion, because the constraint's `expression` contains literals that are bound to the field's type. A field referenced by a `unique` or `primary-key` constraint may apply type promotions that preserve distinctness. Only `name` and `enforced` fields in a constraint definition can be updated, and `timestamp-ms` must be updated when the constraint is updated. The `expression` and `field-ids` fields must not change, as it would change the constraint requirement. Such updates must be done by removing the constraint and adding a new one with a new `constraint-id`. A new `constraint-id` ensures that statuses recorded for the old definition are not applied to the new one. #### Constraint Validation -Enforcement and constraint status are tracked separately. Whether writers must verify the rows that they add is a property of a constraint, tracked by `enforced`. Whether a table is known to satisfy a constraint is a property of a table's data, tracked per snapshot by `constraint-statuses`. +Enforcement and constraint status are tracked separately. Whether writers must verify the rows that they add or modify is a property of a constraint, tracked by `enforced`. Whether a table is known to satisfy a constraint is a property of a table's data, tracked per snapshot by `constraint-statuses`. The status of a constraint for a snapshot is one of: | Status | Description | |---------------|-------------| -| `validated` | The constraint was checked and holds for all rows in the snapshot, or the commit did not add rows and the parent snapshot's status is `validated` | -| `valid` | The parent snapshot's status is `validated` or `valid`, and either the writer can prove that the rows it added follow the constraint or the commit did not add rows. By induction, the constraint holds for all rows in the snapshot. | -| `invalid` | At least one row in the snapshot is known to violate the constraint, either because the constraint was checked or because the parent snapshot's status is `invalid` and the commit did not remove any rows | +| `validated` | The constraint was checked and holds for all rows in the snapshot, or the commit did not add or modify rows and the parent snapshot's status is `validated` | +| `valid` | The parent snapshot's status is `validated` or `valid`, and either the writer can prove that the rows it added or modified follow the constraint or the commit did not add or modify rows. By induction, the constraint holds for all rows in the snapshot. | +| `invalid` | At least one row in the snapshot is known to violate the constraint, either because the constraint was checked or because the parent snapshot's status is `invalid` and the commit did not delete or modify any rows | | `unvalidated` | Whether the constraint holds for all rows in the snapshot is not known | A snapshot's `constraint-statuses` records, for each status, the IDs of the constraints that have that status for the snapshot: @@ -759,15 +761,17 @@ Readers must determine the status of a constraint for a snapshot as follows: 1. If the constraint's `constraint-id` is listed in `validated`, `valid`, `invalid`, or `unvalidated`, that is its status 2. Otherwise, the constraint is `unvalidated` for the snapshot +A `constraint-id` listed in a snapshot that is not in the table's `constraints` refers to a constraint that no longer exists and must be ignored. + Writers must record `constraint-statuses` in every snapshot of a table that has constraints, and must place every constraint that exists when the snapshot is created into exactly one status list. A constraint must not be listed with a status unless the status's description above holds for the snapshot; `unvalidated` is the status of a constraint that cannot be listed with any other status. -A commit that does not add rows cannot introduce a violation. Such a commit, for example a delete or a compaction operation, may carry over the parent snapshot's `validated` or `valid` status for a constraint without checking rows again. +A commit that does not add or modify rows cannot introduce a violation. Such a commit, for example a delete or a compaction operation, may carry over the parent snapshot's `validated` or `valid` status for a constraint without checking rows again. When a constraint becomes enforced, either by being added with `enforced` set to true or by `enforced` changing from false to true, writers should validate the table and record `validated`. A snapshot's `constraint-statuses` must not be modified after the snapshot is created. A `constraint-statuses` update requires a new snapshot. A snapshot that only changes constraint statuses must use the `replace` operation, as it does not add or remove files. -Writers may commit to a table where a constraint is `invalid`. An enforced constraint requires that a writer not add rows that violate the constraint; it does not require a writer to repair existing violations. +Writers may commit to a table where a constraint is `invalid`. An enforced constraint requires that a writer not add or modify rows so that they violate the constraint; it does not require a writer to repair existing violations. ### Manifests @@ -1206,6 +1210,7 @@ A snapshot consists of the following fields: | | | _required_ | **`first-row-id`** | The first `_row_id` assigned to the first row in the first data file in the first manifest, see [Row Lineage](#row-lineage) | | | | _required_ | **`added-rows`** | The upper bound of the number of rows with assigned row IDs, see [Row Lineage](#row-lineage) | | | | _optional_ | **`key-id`** | ID of the encryption key that encrypts the manifest list key metadata | + === "v4" | v4 | Field | Description | | ---------- |------------------------------|-------------| @@ -1226,7 +1231,7 @@ When committed while the table is at version 4 or higher, a snapshot's `timestam The snapshot summary's `operation` field is used by some operations, like snapshot expiration, to skip processing certain snapshots. Possible `operation` values are: * `append` -- Only data files were added and no files were removed. -* `replace` -- Data and delete files were added and removed without changing table data; i.e., compaction, changing the data file format, or relocating data files. A snapshot that only records [constraint statuses](#constraint-validation) and does not add or remove files also uses `replace`. +* `replace` -- Metadata, data, and delete files were added and removed without changing table data; i.e., compaction, changing the data file format, or relocating data files. * `overwrite` -- Data and delete files were added and removed in a logical overwrite operation. * `delete` -- Data files were removed and their contents logically deleted and/or delete files were added to delete rows. @@ -1954,12 +1959,12 @@ Hash results are not dependent on decimal scale, which is part of the type, not Schemas are serialized as a JSON object with the same fields as a struct in the table below, and the following additional fields: -| v1 | v2 and v3 |Field|JSON representation|Example| -| ---------- | ---------- |--- |--- |--- | -| _optional_ | _required_ |**`schema-id`**|`JSON int`|`0`| -| _optional_ | _optional_ |**`identifier-field-ids`**|`JSON list of ints`|`[1, 2]`| +| v1 | v2 and v3 | v4 |Field|JSON representation|Example| +| ---------- | ---------- | ---------- |--- |--- |--- | +| _optional_ | _required_ | _required_ |**`schema-id`**|`JSON int`|`0`| +| _optional_ | _optional_ | _removed_ |**`identifier-field-ids`**|`JSON list of ints`|`[1, 2]`| -`identifier-field-ids` must not be used in v4, where a [`primary-key` constraint](#constraints) expresses the same concept. +In v4, [`primary-key` constraint](#constraints) replaces `identifier-field-ids`. Types are serialized according to this table: @@ -2227,7 +2232,6 @@ Equality deletes are prohibited in v4. Constraints are added in v4: -* [Constraints](#constraints) must not be added to v3 or earlier tables * Table metadata may contain `constraints` and `last-constraint-id` * Snapshots may contain `constraint-statuses` * Upgrading a v2 or v3 table to v4 rewrites the table's `identifier-field-ids` as a `primary-key` constraint that is not enforced, unless a field cannot be used in a `primary-key` constraint From 06f8e2099c369f6cf1e08e23aa82799a3ebcabba Mon Sep 17 00:00:00 2001 From: Huaxin Gao Date: Wed, 21 Oct 2026 11:47:36 -0700 Subject: [PATCH 24/25] Remove unvalidated rule for check constraints that call a UDF --- format/spec.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/format/spec.md b/format/spec.md index 1f944f0daad8..64b22056bff7 100644 --- a/format/spec.md +++ b/format/spec.md @@ -720,8 +720,6 @@ A check expression is evaluated for each row over the values of that row. An exp A check expression must produce the same result every time it is evaluated for the same row. A function that depends on anything other than its arguments, such as the current time or a random value, must not be called. A [user-defined function](udf-spec.md) must not be called unless its current definition version declares `deterministic` as true. A user-defined function is a function loaded from a catalog as described in the UDF spec. Functions in the reserved `sql_functions` and `iceberg_functions` sets and engine built-in functions, as described in the [expressions spec](expressions-spec.md#apply-function), are not user-defined functions. -A function definition can change without a table commit, and `constraint-statuses` does not record which function definitions were used, so a status for a `check` constraint that calls a user-defined function could become wrong without any write. Such a constraint must always be listed as `unvalidated`. When it is enforced, writers still evaluate it with the current function definitions for the rows that they add or modify. - #### Constraints and Schema Evolution A constraint references fields by ID, so schema changes interact with constraints as follows. The referenced fields of a `check` constraint are the field IDs in its `expression`; the referenced fields of a `unique` or `primary-key` constraint are its `field-ids`. From 78a79714bc81c6253d42178c7fc74408c8940724 Mon Sep 17 00:00:00 2001 From: Huaxin Gao Date: Wed, 21 Oct 2026 11:53:11 -0700 Subject: [PATCH 25/25] Assume UDFs do not change in check constraints --- format/spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/format/spec.md b/format/spec.md index 64b22056bff7..1791e1f7c9a2 100644 --- a/format/spec.md +++ b/format/spec.md @@ -718,7 +718,7 @@ A check expression is evaluated for each row over the values of that row. An exp * Must not depend on another table, such as subqueries * Must not reference a field within a `list` or a `map`, because such a field has a value for each element rather than one value for the row; it may pass a whole `list` or `map` field to a function, for example to check that a list is not empty -A check expression must produce the same result every time it is evaluated for the same row. A function that depends on anything other than its arguments, such as the current time or a random value, must not be called. A [user-defined function](udf-spec.md) must not be called unless its current definition version declares `deterministic` as true. A user-defined function is a function loaded from a catalog as described in the UDF spec. Functions in the reserved `sql_functions` and `iceberg_functions` sets and engine built-in functions, as described in the [expressions spec](expressions-spec.md#apply-function), are not user-defined functions. +A check expression must produce the same result every time it is evaluated for the same row. A function that depends on anything other than its arguments, such as the current time or a random value, must not be called. A [user-defined function](udf-spec.md) must not be called unless its current definition version declares `deterministic` as true. A user-defined function is a function loaded from a catalog as described in the UDF spec. Functions in the reserved `sql_functions` and `iceberg_functions` sets and engine built-in functions, as described in the [expressions spec](expressions-spec.md#apply-function), are not user-defined functions. The behavior of a user-defined function is assumed not to change. A writer or a reader is not required to detect a change to a user-defined function or to recompute `constraint-statuses` when the function changes. #### Constraints and Schema Evolution