Skip to content

Commit

Permalink
Patch 10 (#6368)
Browse files Browse the repository at this point in the history
## What are you changing in this pull request and why?
<!--
Describe your changes and why you're making them. If related to an open
issue or a pull request on dbt Core or another repository, then link to
them here!

To learn more about the writing conventions used in the dbt Labs docs,
see the [Content style
guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md).
-->
Adding documentation for Databricks Behavior Flags

## Checklist
- [x] I have reviewed the [Content style
guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md)
so my content adheres to these guidelines.
- [x] The topic I'm writing about is for specific dbt version(s) and I
have versioned it according to the [version a whole
page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version)
and/or [version a block of
content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content)
guidelines.
- [x] I have added checklist item(s) to this list for anything anything
that needs to happen before this PR is merged, such as "needs technical
review" or "change base branch."
- [x] The content in this PR requires a dbt release note, so I added one
to the [release notes
page](https://docs.getdbt.com/docs/dbt-versions/dbt-cloud-release-notes).
<!--
PRE-RELEASE VERSION OF dbt (if so, uncomment):
- [ ] Add a note to the prerelease version [Migration
Guide](https://github.com/dbt-labs/docs.getdbt.com/tree/current/website/docs/docs/dbt-versions/core-upgrade)
-->
ADDING OR REMOVING PAGES (if so, uncomment):
- [x] Add/remove page in `website/sidebars.js`
- [x] Provide a unique filename for new pages

---------

Co-authored-by: Leona B. Campbell <[email protected]>
Co-authored-by: Amy Chen <[email protected]>
Co-authored-by: Matt Shaver <[email protected]>
  • Loading branch information
4 people authored Oct 31, 2024
1 parent 7004dc5 commit d5e48f1
Show file tree
Hide file tree
Showing 3 changed files with 35 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,17 @@ Some adapters can display behavior changes when certain flags are enabled. The f

<div className="grid--3-col">

<Card
title="Databricks"
body="Behavior changes for the Databricks adapter."
link="reference/global-configs/databricks-changes"
icon="databricks"/>


<Card
title="Redshift"
body="Behavior changes for the Amazon Redshift adapter."
link="reference/global-configs/redshift-changes"
icon="redshift"/>

</div>
</div>
26 changes: 26 additions & 0 deletions website/docs/reference/global-configs/databricks-changes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
---
title: "Databricks adapter behavior changes"
id: "databricks-changes"
sidebar: "Databricks"
---

The following are the current [behavior change flags](/docs/reference/global-configs/behavior-changes.md#behavior-change-flags) that are specific to `dbt-databricks`:

| Flag | `dbt-databricks`: Intro | `dbt-databricks`: Maturity |
| ----------------------------- | ----------------------- | -------------------------- |
| `use_info_schema_for_columns` | 1.9.0 | TBD |
| `use_user_folder_for_python` | 1.9.0 | TBD |

### Use information schema for columns

The `use_info_schema_for_columns` flag is `False` by default.

Setting this flag to `True` will use `information_schema` rather than `describe extended` to get column metadata for Unity Catalog tables. This setting helps you avoid issues where `describe extended` truncates information when the type is a complex struct. However, this setting is not yet the default behavior, as there are performance impacts due to a Databricks metadata limitation because of the need to run `REPAIR TABLE {{relation}} SYNC METADATA` before querying to ensure the `information_schema` is complete.

This flag will become the default behavior when this additional query is no longer needed.

### Use user's folder for Python model notebooks

The `use_user_folder_for_python` flag is `False` by default and results in writing uploaded python model notebooks to `/Shared/dbt_python_models/{{schema}}/`. Setting this flag to `True` will write notebooks to `/Users/{{current user}}/{{catalog}}/{{schema}}/` Writing to the `Shared` folder is deprecated by Databricks as it does not align with governance best practices.

We plan to promote this flag to maturity in v1.10.0.
1 change: 1 addition & 0 deletions website/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -1104,6 +1104,7 @@ const sidebarSettings = {
link: { type: "doc", id: "reference/global-configs/adapter-behavior-changes" },
items: [
"reference/global-configs/adapter-behavior-changes",
"reference/global-configs/databricks-changes",
"reference/global-configs/redshift-changes",
],
},
Expand Down

0 comments on commit d5e48f1

Please sign in to comment.