From c51c816fe669e148ca5ad3837324175abdb7d83a Mon Sep 17 00:00:00 2001 From: "Leona B. Campbell" <3880403+runleonarun@users.noreply.github.com> Date: Fri, 17 Jan 2025 13:31:58 -0500 Subject: [PATCH] Move limitation to more appropriate location for athena configs (#6786) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## What are you changing in this pull request and why? ## Checklist - [ ] I have reviewed the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ ] The topic I'm writing about is for specific dbt version(s) and I have versioned it according to the [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and/or [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content) guidelines. - [ ] I have added checklist item(s) to this list for anything anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." - [ ] The content in this PR requires a dbt release note, so I added one to the [release notes page](https://docs.getdbt.com/docs/dbt-versions/dbt-cloud-release-notes). --- 🚀 Deployment available! Here are the direct links to the updated files: - https://docs-getdbt-com-git-runleonarun-patch-16-dbt-labs.vercel.app/reference/resource-configs/athena-configs --------- Co-authored-by: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> --- .../docs/reference/resource-configs/athena-configs.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/website/docs/reference/resource-configs/athena-configs.md b/website/docs/reference/resource-configs/athena-configs.md index fd5bc663ee7..082f3b5c249 100644 --- a/website/docs/reference/resource-configs/athena-configs.md +++ b/website/docs/reference/resource-configs/athena-configs.md @@ -106,7 +106,7 @@ lf_grants={ -There are some limitations and recommendations that should be considered: +Consider these limitations and recommendations: - `lf_tags` and `lf_tags_columns` configs support only attaching lf tags to corresponding resources. - We recommend managing LF Tags permissions somewhere outside dbt. For example, [terraform](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lakeformation_permissions) or [aws cdk](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_lakeformation-readme.html). @@ -114,8 +114,7 @@ There are some limitations and recommendations that should be considered: - Any tags listed in `lf_inherited_tags` should be strictly inherited from the database level and never overridden at the table and column level. - Currently, `dbt-athena` does not differentiate between an inherited tag association and an override it made previously. - For example, If a `lf_tags_config` value overrides an inherited tag in one run, and that override is removed before a subsequent run, the prior override will linger and no longer be encoded anywhere (for example, Terraform where the inherited value is configured nor in the DBT project where the override previously existed but now is gone). - - + ### Table location The saved location of a table is determined in precedence by the following conditions: @@ -144,6 +143,9 @@ The following [incremental models](https://docs.getdbt.com/docs/build/incrementa - `append`: Insert new records without updating, deleting or overwriting any existing data. There might be duplicate data (great for log or historical data). - `merge`: Conditionally updates, deletes, or inserts rows into an Iceberg table. Used in combination with `unique_key`.It is only available when using Iceberg. +Consider this limitation when using Iceberg models: + +- Incremental Iceberg models — Sync all columns on schema change. You can't remove columns used for partitioning with an incremental refresh; you must fully refresh the model. ### On schema change @@ -361,8 +363,7 @@ The materialization also supports invalidating hard deletes. For usage details, ### Snapshots known issues -- Incremental Iceberg models - Sync all columns on schema change. Columns used for partitioning can't be removed. From a dbt perspective, the only way is to fully refresh the incremental model. -- Tables, schemas and database names should only be lowercase +- Tables, schemas, and database names should only be lowercase. - To avoid potential conflicts, make sure [`dbt-athena-adapter`](https://github.com/Tomme/dbt-athena) is not installed in the target environment. - Snapshot does not support dropping columns from the source table. If you drop a column, make sure to drop the column from the snapshot as well. Another workaround is to NULL the column in the snapshot definition to preserve the history.