From eb93839faec4ef5fad5fa113d0f38d87e635fbb3 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Wed, 6 Nov 2024 11:54:57 +0000 Subject: [PATCH 01/81] add tags to exports --- website/docs/docs/build/saved-queries.md | 2 + .../docs/docs/dbt-versions/release-notes.md | 8 ++++ .../docs/use-dbt-semantic-layer/exports.md | 10 +++-- .../docs/reference/resource-configs/tags.md | 37 ++++++++++++++++++- 4 files changed, 53 insertions(+), 4 deletions(-) diff --git a/website/docs/docs/build/saved-queries.md b/website/docs/docs/build/saved-queries.md index ed56d13dcc9..d1fe785f483 100644 --- a/website/docs/docs/build/saved-queries.md +++ b/website/docs/docs/build/saved-queries.md @@ -58,6 +58,7 @@ Note that we use the double colon (::) to indicate whether a parameter is nested | `exports::config::export_as` | String | Required | Specifies the type of export: table, view, or upcoming cache options. Nested within `exports` and `config`. | | `exports::config::schema` | String | Optional | Schema for creating the table or view, not applicable for caching. Nested within `exports` and `config`. | | `exports::config::alias` | String | Optional | Table alias used to write to the table or view. This option can't be used for caching. Nested within `exports` and `config`. | +| `exports::config::tag` | String | Optional | Apply a [tag](/reference/resource-configs/tags) (or list of tags) to a resource. Tags help organize and filter resources in dbt. Nested within `exports` and `config`. | @@ -99,6 +100,7 @@ saved_queries: alias: my_export_alias export_as: table schema: my_export_schema_name + [tags](/reference/resource-configs/tags): ['my_tag'] ``` diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index c3f0bbfbe06..54326474e7c 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -24,6 +24,14 @@ Release notes are grouped by month for both multi-tenant and virtual private clo - Improved handling of queries when multiple tables are selected in a data source. - Fixed a bug when an IN filter contained a lot of values. - Better error messaging for queries that can't be parsed correctly. +- **New**: Exports now support [tags](/docs/reference/resource-configs/tags) in dbt Cloud. Tags allow you to categorize your resources and filter them in dbt Cloud. You can add tags to your [exports](/docs/build/saved-queries#configure-exports) in the `semantic_model.yml` file or `dbt_project.yml` file. For example: + + ```yml + exports: + - name: export_name + tags: ['export_tag'] + ... + ``` ## October 2024 diff --git a/website/docs/docs/use-dbt-semantic-layer/exports.md b/website/docs/docs/use-dbt-semantic-layer/exports.md index 5d6e4c0d996..af8e0dce1cd 100644 --- a/website/docs/docs/use-dbt-semantic-layer/exports.md +++ b/website/docs/docs/use-dbt-semantic-layer/exports.md @@ -29,16 +29,20 @@ The following section explains the main benefits of using exports, including: - [Easier changes](#easier-changes) - [Caching](#caching) -#### DRY representation + Currently, creating tables often involves generating tens, hundreds, or even thousands of tables that denormalize data into summary or metric mart tables. The main benefit of exports is creating a "Don't Repeat Yourself (DRY)" representation of the logic to construct each metric, dimension, join, filter, and so on. This allows you to reuse those components for long-term scalability, even if you're replacing manually written SQL models with references to the metrics or dimensions in saved queries. + -#### Easier changes + Exports ensure that changes to metrics and dimensions are made in one place and then cascade to those various destinations seamlessly. This prevents the problem of needing to update a metric across every model that references that same concept. + + + -#### Caching Use exports to pre-populate the cache, so that you're pre-computing what you need to serve users through the dynamic Semantic Layer APIs. + #### Considerations diff --git a/website/docs/reference/resource-configs/tags.md b/website/docs/reference/resource-configs/tags.md index f6c46f8a088..75d5c55a1af 100644 --- a/website/docs/reference/resource-configs/tags.md +++ b/website/docs/reference/resource-configs/tags.md @@ -30,6 +30,10 @@ seeds: [](/reference/resource-configs/resource-path): +tags: | [] +[exports](/docs/use-dbt-semantic-layer/exports): + [](/reference/resource-configs/resource-path): + +tags: | [] + ``` @@ -164,6 +168,32 @@ seeds: +### Apply tags to exports + + + +```yml +[exports](/docs/use-dbt-semantic-layer/exports): + jaffle_shop: + customer_order_metrics: + +tags: order_metrics +``` + + + + + +```yml +[exports](/docs/use-dbt-semantic-layer/exports): + jaffle_shop: + customer_order_metrics: + +tags: + - order_metrics + - hourly +``` + + + ## Usage notes ### Tags are additive @@ -178,7 +208,7 @@ Tags accumulate hierarchically. The above example would result in: ### Other resource types -Tags can also be applied to sources, exposures, and even _specific columns_ in a resource. +Tags can also be applied to [sources](/docs/build/sources), [exposures](/docs/build/exposures), [exports](/docs/use-dbt-semantic-layer/exports), and even _specific columns_ in a resource. These resources do not yet support the `config` property, so you'll need to specify the tags as a top-level key instead. @@ -206,6 +236,11 @@ sources: tests: - unique: tags: ['test_level'] + +exports: + - name: export_name + tags: ['export_tag'] + ... ``` From 744352134d1761094eb7e68a1f6abe34ba76c4af Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Wed, 6 Nov 2024 12:02:33 +0000 Subject: [PATCH 02/81] fix link --- website/docs/docs/build/saved-queries.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/saved-queries.md b/website/docs/docs/build/saved-queries.md index d1fe785f483..db978ece436 100644 --- a/website/docs/docs/build/saved-queries.md +++ b/website/docs/docs/build/saved-queries.md @@ -100,7 +100,7 @@ saved_queries: alias: my_export_alias export_as: table schema: my_export_schema_name - [tags](/reference/resource-configs/tags): ['my_tag'] + [tags](/reference/resource-configs/tags): 'my_tag' ``` From 44b92484699a3e07b41634e43f6fef16a2db84b0 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Wed, 6 Nov 2024 12:02:57 +0000 Subject: [PATCH 03/81] fix link --- website/docs/docs/dbt-versions/release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index 54326474e7c..f7faac30645 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -24,7 +24,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo - Improved handling of queries when multiple tables are selected in a data source. - Fixed a bug when an IN filter contained a lot of values. - Better error messaging for queries that can't be parsed correctly. -- **New**: Exports now support [tags](/docs/reference/resource-configs/tags) in dbt Cloud. Tags allow you to categorize your resources and filter them in dbt Cloud. You can add tags to your [exports](/docs/build/saved-queries#configure-exports) in the `semantic_model.yml` file or `dbt_project.yml` file. For example: +- **New**: Exports now support [tags](/reference/resource-configs/tags) in dbt Cloud. Tags allow you to categorize your resources and filter them in dbt Cloud. You can add tags to your [exports](/docs/build/saved-queries#configure-exports) in the `semantic_model.yml` file or `dbt_project.yml` file. For example: ```yml exports: From 3571c0f0ec44aac230cbd7680962bc8be8db163d Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Wed, 6 Nov 2024 12:43:24 +0000 Subject: [PATCH 04/81] add meta --- website/docs/docs/build/dimensions.md | 7 +++ .../docs/docs/dbt-versions/release-notes.md | 1 + .../docs/reference/resource-configs/meta.md | 52 +++++++++++++++++++ 3 files changed, 60 insertions(+) diff --git a/website/docs/docs/build/dimensions.md b/website/docs/docs/build/dimensions.md index 5026f4c45cd..eb1809ea72b 100644 --- a/website/docs/docs/build/dimensions.md +++ b/website/docs/docs/build/dimensions.md @@ -22,6 +22,7 @@ All dimensions require a `name`, `type`, and can optionally include an `expr` pa | `description` | A clear description of the dimension | Optional | | `expr` | Defines the underlying column or SQL query for a dimension. If no `expr` is specified, MetricFlow will use the column with the same name as the group. You can use the column name itself to input a SQL expression. | Optional | | `label` | A recommended string that defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | +| `meta` | Use the [`meta` field](/reference/resource-configs/meta) to set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | Refer to the following for the complete specification for dimensions: @@ -59,6 +60,8 @@ semantic_models: type_params: time_granularity: day label: "Date of transaction" # Recommend adding a label to provide more context to users consuming the data + meta: + data_owner: "Finance team" expr: ts - name: is_bulk type: categorical @@ -106,6 +109,8 @@ dimensions: - name: is_bulk_transaction type: categorical expr: case when quantity > 10 then true else false end + meta: + usage: "Filter to identify bulk transactions, like where quantity > 10." ``` ## Time @@ -136,6 +141,8 @@ dimensions: type: time label: "Date of creation" expr: ts_created # ts_created is the underlying column name from the table + meta: + notes: "Only valid for orders from 2022 onward" is_partition: True type_params: time_granularity: day diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index c3f0bbfbe06..8dd16e99bbb 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -24,6 +24,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo - Improved handling of queries when multiple tables are selected in a data source. - Fixed a bug when an IN filter contained a lot of values. - Better error messaging for queries that can't be parsed correctly. +- **New**: [Dimensions](/reference/resource-configs/meta) now support the meta config property in dbt Cloud Versionless ??? or v1.9+??. You can add metadata to your dimensions to provide additional context and information about the dimension. Refer to [meta](/reference/resource-configs/meta) for more information. ## October 2024 diff --git a/website/docs/reference/resource-configs/meta.md b/website/docs/reference/resource-configs/meta.md index 53a4f77184e..ce33feff62e 100644 --- a/website/docs/reference/resource-configs/meta.md +++ b/website/docs/reference/resource-configs/meta.md @@ -19,6 +19,7 @@ hide_table_of_contents: true { label: 'Semantic Models', value: 'semantic models', }, { label: 'Metrics', value: 'metrics', }, { label: 'Saved queries', value: 'saved queries', }, + { label: 'Dimensions', value: 'dimensions', }, ] }> @@ -250,6 +251,22 @@ saved_queries: + + + + +```yml +semantic_models: + - name: semantic_model + ... + dimensions: + - name: order_date + config: + meta: {} +``` + + + @@ -343,3 +360,38 @@ models: +### Assign owner and additional metadata to dimensions + +The following example shows how to assign a `data_owner` and additional metadata value to a dimension in a semantic model. + + + +```yml +semantic_models: + - name: semantic_model + ... + dimensions: + - name: order_date + meta: + data_owner: "Finance team" + used_in_reporting: true +``` + + + +This second example shows how to assign a `data_owner` and additional metadata value to a dimension in the `dbt_project.yml` file using the `+meta` syntax. + + + +```yml +semantic-models: + jaffle_shop: + ... + dimensions: + - name: order_date + +meta: + data_owner: "Finance team" + used_in_reporting: true +``` + + From 53c0351a95a72f6212768191b24ede084e48717b Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Wed, 6 Nov 2024 12:51:35 +0000 Subject: [PATCH 05/81] update example --- website/docs/docs/build/saved-queries.md | 2 +- .../docs/reference/resource-configs/tags.md | 28 +++++++++++++------ 2 files changed, 20 insertions(+), 10 deletions(-) diff --git a/website/docs/docs/build/saved-queries.md b/website/docs/docs/build/saved-queries.md index db978ece436..136474da3ee 100644 --- a/website/docs/docs/build/saved-queries.md +++ b/website/docs/docs/build/saved-queries.md @@ -58,7 +58,7 @@ Note that we use the double colon (::) to indicate whether a parameter is nested | `exports::config::export_as` | String | Required | Specifies the type of export: table, view, or upcoming cache options. Nested within `exports` and `config`. | | `exports::config::schema` | String | Optional | Schema for creating the table or view, not applicable for caching. Nested within `exports` and `config`. | | `exports::config::alias` | String | Optional | Table alias used to write to the table or view. This option can't be used for caching. Nested within `exports` and `config`. | -| `exports::config::tag` | String | Optional | Apply a [tag](/reference/resource-configs/tags) (or list of tags) to a resource. Tags help organize and filter resources in dbt. Nested within `exports` and `config`. | +| `exports::config::tags` | String | Optional | Apply a [tag](/reference/resource-configs/tags) (or list of tags) to a resource. Tags help organize and filter resources in dbt. Nested within `exports` and `config`. | diff --git a/website/docs/reference/resource-configs/tags.md b/website/docs/reference/resource-configs/tags.md index 75d5c55a1af..6018c774282 100644 --- a/website/docs/reference/resource-configs/tags.md +++ b/website/docs/reference/resource-configs/tags.md @@ -170,10 +170,12 @@ seeds: ### Apply tags to exports +This following example shows how to apply tags to an export in the `dbt_project.yml` file. The export is then tagged with `order_metrics`. + ```yml -[exports](/docs/use-dbt-semantic-layer/exports): +[exports](/docs/build/saved-queries-configure-exports): jaffle_shop: customer_order_metrics: +tags: order_metrics @@ -181,15 +183,23 @@ seeds: - +The second example shows how to apply tags to an export in the `semantic_model.yml` file. The export is then tagged with `order_metrics` and `hourly`. -```yml -[exports](/docs/use-dbt-semantic-layer/exports): - jaffle_shop: - customer_order_metrics: - +tags: - - order_metrics - - hourly + + +```yaml +saved_queries: + - name: order_metrics + ... + exports: + - name: hourly_order_metrics + config: + alias: my_export_alias + export_as: table + schema: my_export_schema_name + tags: + - order_metrics + - hourly ``` From 2d5a67592296524f884b5097f7b2be932fd367b6 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Wed, 6 Nov 2024 12:52:56 +0000 Subject: [PATCH 06/81] add link --- website/docs/reference/resource-configs/meta.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/docs/reference/resource-configs/meta.md b/website/docs/reference/resource-configs/meta.md index ce33feff62e..fa0d48c299e 100644 --- a/website/docs/reference/resource-configs/meta.md +++ b/website/docs/reference/resource-configs/meta.md @@ -259,7 +259,7 @@ saved_queries: semantic_models: - name: semantic_model ... - dimensions: + [dimensions](/docs/build/dimensions): - name: order_date config: meta: {} @@ -370,7 +370,7 @@ The following example shows how to assign a `data_owner` and additional metadata semantic_models: - name: semantic_model ... - dimensions: + [dimensions](/docs/build/dimensions): - name: order_date meta: data_owner: "Finance team" @@ -387,7 +387,7 @@ This second example shows how to assign a `data_owner` and additional metadata v semantic-models: jaffle_shop: ... - dimensions: + [dimensions](/docs/build/dimensions): - name: order_date +meta: data_owner: "Finance team" From c9a08209e458134e607a19811527bcf43a475eab Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Wed, 6 Nov 2024 12:58:11 +0000 Subject: [PATCH 07/81] fix link --- website/docs/reference/resource-configs/tags.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/reference/resource-configs/tags.md b/website/docs/reference/resource-configs/tags.md index 6018c774282..992cc1ea22a 100644 --- a/website/docs/reference/resource-configs/tags.md +++ b/website/docs/reference/resource-configs/tags.md @@ -30,7 +30,7 @@ seeds: [](/reference/resource-configs/resource-path): +tags: | [] -[exports](/docs/use-dbt-semantic-layer/exports): +[exports](/docs/build/saved-queries#configure-exports): [](/reference/resource-configs/resource-path): +tags: | [] @@ -175,7 +175,7 @@ This following example shows how to apply tags to an export in the `dbt_project. ```yml -[exports](/docs/build/saved-queries-configure-exports): +[exports](/docs/build/saved-queries#configure-exports): jaffle_shop: customer_order_metrics: +tags: order_metrics From 8162530983af7e1be9457e7743d8c02993294ed1 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Mon, 18 Nov 2024 09:40:23 -0500 Subject: [PATCH 08/81] update tags --- website/docs/docs/build/saved-queries.md | 116 +++++++++++++++--- .../docs/reference/resource-configs/tags.md | 44 +++++-- 2 files changed, 138 insertions(+), 22 deletions(-) diff --git a/website/docs/docs/build/saved-queries.md b/website/docs/docs/build/saved-queries.md index 136474da3ee..68876b65160 100644 --- a/website/docs/docs/build/saved-queries.md +++ b/website/docs/docs/build/saved-queries.md @@ -17,8 +17,33 @@ To create a saved query, refer to the following table parameters. :::tip Note that we use the double colon (::) to indicate whether a parameter is nested within another parameter. So for example, `query_params::metrics` means the `metrics` parameter is nested under `query_params`. ::: + + + + +| Parameter | Type | Required | Description | +|-------|---------|----------|----------------| +| `name` | String | Required | Name of the saved query object. | +| `description` | String | Required | A description of the saved query. | +| `label` | String | Required | The display name for your saved query. This value will be shown in downstream tools. | +| `config` | String | Optional | Use the [`config`](/reference/resource-properties/config) property to specify configurations for your saved query. Supports `cache`, [`enabled`](/reference/resource-configs/enabled), `export_as`, [`group`](/reference/resource-configs/group), [`meta`](/reference/resource-configs/meta), and [`schema`](/reference/resource-configs/schema) configurations. | +| `config::cache::enabled` | Object | Optional | An object with a sub-key used to specify if a saved query should populate the [cache](/docs/use-dbt-semantic-layer/sl-cache). Accepts sub-key `true` or `false`. Defaults to `false` | +| `query_params` | Structure | Required | Contains the query parameters. | +| `query_params::metrics` | List or String | Optional | A list of the metrics to be used in the query as specified in the command line interface. | +| `query_params::group_by` | List or String | Optional | A list of the Entities and Dimensions to be used in the query, which include the `Dimension` or `TimeDimension`. | +| `query_params::where` | List or String | Optional | A list of strings that may include the `Dimension` or `TimeDimension` objects. | +| `exports` | List or Structure | Optional | A list of exports to be specified within the exports structure. | +| `exports::name` | String | Required | Name of the export object. | +| `exports::config` | List or Structure | Required | A [`config`](/reference/resource-properties/config) property for any parameters specifying the export. | +| `exports::config::export_as` | String | Required | The type of export to run. Options include table or view currently and cache in the near future. | +| `exports::config::schema` | String | Optional | The [schema](/reference/resource-configs/schema) for creating the table or view. This option cannot be used for caching. | +| `exports::config::alias` | String | Optional | The table [alias](/reference/resource-configs/alias) used to write to the table or view. This option cannot be used for caching. | +| `exports::config::tags` | String | Optional | Apply a [tag](/reference/resource-configs/tags) (or list of tags) to a resource. Tags help organize and filter resources in dbt. Nested within `exports` and `config`. | + + + - + | Parameter | Type | Required | Description | |-------|---------|----------|----------------| @@ -33,15 +58,15 @@ Note that we use the double colon (::) to indicate whether a parameter is nested | `query_params::where` | List or String | Optional | A list of strings that may include the `Dimension` or `TimeDimension` objects. | | `exports` | List or Structure | Optional | A list of exports to be specified within the exports structure. | | `exports::name` | String | Required | Name of the export object. | -| `exports::config` | List or Structure | Required | A config section for any parameters specifying the export. | +| `exports::config` | List or Structure | Required | A [`config`](/reference/resource-properties/config) property for any parameters specifying the export. | | `exports::config::export_as` | String | Required | The type of export to run. Options include table or view currently and cache in the near future. | | `exports::config::schema` | String | Optional | The schema for creating the table or view. This option cannot be used for caching. | -| `exports::config::alias` | String | Optional | The table alias used to write to the table or view. This option cannot be used for caching. | +| `exports::config::alias` | String | Optional | The table alias used to write to the table or view. This option cannot be used for caching. | - + | Parameter | Type | Required | Description | |-------|---------|----------|----------------| @@ -54,11 +79,10 @@ Note that we use the double colon (::) to indicate whether a parameter is nested | `query_params::where` | List or String | Optional | Conditions nested with the `query_params`: a list of strings that may include the `Dimension` or `TimeDimension` objects. | | `exports` | List or Structure | Optional | A list of exports to be specified within the exports structure. | | `exports::name` | String | Required | Name of export object, nested within `exports`. | -| `exports::config` | List or Structure | Required | A config section for any parameters specifying the export, nested within `exports`. | +| `exports::config` | List or Structure | Required | A [`config`](/reference/resource-properties/config) property for any parameters specifying the export, nested within `exports`. | | `exports::config::export_as` | String | Required | Specifies the type of export: table, view, or upcoming cache options. Nested within `exports` and `config`. | | `exports::config::schema` | String | Optional | Schema for creating the table or view, not applicable for caching. Nested within `exports` and `config`. | | `exports::config::alias` | String | Optional | Table alias used to write to the table or view. This option can't be used for caching. Nested within `exports` and `config`. | -| `exports::config::tags` | String | Optional | Apply a [tag](/reference/resource-configs/tags) (or list of tags) to a resource. Tags help organize and filter resources in dbt. Nested within `exports` and `config`. | @@ -70,13 +94,12 @@ Use saved queries to define and manage common Semantic Layer queries in YAML, in In your saved query config, you can also leverage [caching](/docs/use-dbt-semantic-layer/sl-cache) with the dbt Cloud job scheduler to cache common queries, speed up performance, and reduce compute costs. - - - + In the following example, you can set the saved query in the `semantic_model.yml` file: + ```yaml saved_queries: @@ -97,13 +120,45 @@ saved_queries: exports: - name: my_export config: + export_as: table alias: my_export_alias + schema: my_export_schema_name + tags: 'my_tag' +``` + + + + + +```yaml +saved_queries: + - name: test_saved_query + description: "{{ doc('saved_query_description') }}" + label: Test saved query + config: + cache: + enabled: true # Or false if you want it disabled by default + query_params: + metrics: + - simple_metric + group_by: + - "Dimension('user__ds')" + where: + - "{{ Dimension('user__ds', 'DAY') }} <= now()" + - "{{ Dimension('user__ds', 'DAY') }} >= '2023-01-01'" + exports: + - name: my_export + config: export_as: table + alias: my_export_alias schema: my_export_schema_name - [tags](/reference/resource-configs/tags): 'my_tag' ``` + + + + Note, that you can set `export_as` to both the saved query and the exports [config](/reference/resource-properties/config), with the exports config value taking precedence. If a key isn't set in the exports config, it will inherit the saved query config value. #### Where clause @@ -123,7 +178,6 @@ filter: | filter: | {{ Metric('metric_name', group_by=['entity_name']) }} ``` - @@ -149,8 +203,8 @@ saved_queries: exports: - name: my_export config: - alias: my_export_alias export_as: table + alias: my_export_alias schema: my_export_schema_name ``` @@ -183,6 +237,38 @@ Once you've configured your saved query and set the foundation block, you can no The following is an example of a saved query with an export: + + +```yaml +saved_queries: + - name: order_metrics + description: Relevant order metrics + query_params: + metrics: + - orders + - large_order + - food_orders + - order_total + group_by: + - Entity('order_id') + - TimeDimension('metric_time', 'day') + - Dimension('customer__customer_name') + - ... # Additional group_by + where: + - "{{TimeDimension('metric_time')}} > current_timestamp - interval '1 week'" + - ... # Additional where clauses + exports: + - name: order_metrics + config: + export_as: table # Options available: table, view + [alias](/reference/resource-configs/alias): my_export_alias # Optional - defaults to Export name + [schema](/reference/resource-configs/schema): my_export_schema_name # Optional - defaults to deployment schema + [tags](/reference/resource-configs/tags): 'my_tag' + +``` + + + ```yaml saved_queries: @@ -206,9 +292,11 @@ saved_queries: - name: order_metrics config: export_as: table # Options available: table, view - schema: YOUR_SCHEMA # Optional - defaults to deployment schema - alias: SOME_TABLE_NAME # Optional - defaults to Export name + schema: my_export_schema_name # Optional - defaults to deployment schema + alias: my_export_alias # Optional - defaults to Export name ``` + + ## Run exports diff --git a/website/docs/reference/resource-configs/tags.md b/website/docs/reference/resource-configs/tags.md index 992cc1ea22a..e8838a72422 100644 --- a/website/docs/reference/resource-configs/tags.md +++ b/website/docs/reference/resource-configs/tags.md @@ -16,6 +16,8 @@ datatype: string | [string] + + ```yml models: @@ -30,11 +32,32 @@ seeds: [](/reference/resource-configs/resource-path): +tags: | [] -[exports](/docs/build/saved-queries#configure-exports): +``` + + + + +```yml + +models: + [](/reference/resource-configs/resource-path): + +tags: | [] + +snapshots: + [](/reference/resource-configs/resource-path): + +tags: | [] + +seeds: + [](/reference/resource-configs/resource-path): + +tags: | [] + +exports: [](/reference/resource-configs/resource-path): +tags: | [] ``` + + @@ -170,6 +193,14 @@ seeds: ### Apply tags to exports + + +Applying tags to exports is only available in dbt Core versions 1.9 and later. + + + + + This following example shows how to apply tags to an export in the `dbt_project.yml` file. The export is then tagged with `order_metrics`. @@ -203,11 +234,12 @@ saved_queries: ``` + ## Usage notes ### Tags are additive -Tags accumulate hierarchically. The above example would result in: +Tags accumulate hierarchically. The [earlier example](/reference/resource-configs/tags#use-tags-to-run-parts-of-your-project) would result in: | Model | Tags | | -------------------------------- | ------------------------------------- | @@ -218,7 +250,7 @@ Tags accumulate hierarchically. The above example would result in: ### Other resource types -Tags can also be applied to [sources](/docs/build/sources), [exposures](/docs/build/exposures), [exports](/docs/use-dbt-semantic-layer/exports), and even _specific columns_ in a resource. +Tags can also be applied to [sources](/docs/build/sources), [exposures](/docs/build/exposures), and even _specific columns_ in a resource. These resources do not yet support the `config` property, so you'll need to specify the tags as a top-level key instead. @@ -246,15 +278,11 @@ sources: tests: - unique: tags: ['test_level'] - -exports: - - name: export_name - tags: ['export_tag'] - ... ``` + In the example above, the `unique` test would be selected by any of these four tags: ```bash $ dbt test --select tag:top_level From 4cd118c64635203116552b90fe2c7c9f8ea1373b Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Mon, 18 Nov 2024 09:42:56 -0500 Subject: [PATCH 09/81] update rn --- website/docs/docs/dbt-versions/release-notes.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index f7faac30645..d96d5207bd3 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -19,12 +19,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo \* The official release date for this new format of release notes is May 15th, 2024. Historical release notes for prior dates may not reflect all available features released earlier this year or their tenancy availability. ## November 2024 -- **Fix**: This update improves [dbt Semantic Layer Tableau integration](/docs/cloud-integrations/semantic-layer/tableau) making query parsing more reliable. Some key fixes include: - - Error messages for unsupported joins between saved queries and ALL tables. - - Improved handling of queries when multiple tables are selected in a data source. - - Fixed a bug when an IN filter contained a lot of values. - - Better error messaging for queries that can't be parsed correctly. -- **New**: Exports now support [tags](/reference/resource-configs/tags) in dbt Cloud. Tags allow you to categorize your resources and filter them in dbt Cloud. You can add tags to your [exports](/docs/build/saved-queries#configure-exports) in the `semantic_model.yml` file or `dbt_project.yml` file. For example: +- **New**: Exports now support [tags](/reference/resource-configs/tags) in dbt. Tags allow you to categorize your resources and filter them. You can add tags to your [exports](/docs/build/saved-queries#configure-exports) in the `semantic_model.yml` file or `dbt_project.yml` file. For example: ```yml exports: @@ -32,6 +27,12 @@ Release notes are grouped by month for both multi-tenant and virtual private clo tags: ['export_tag'] ... ``` +- **Fix**: This update improves [dbt Semantic Layer Tableau integration](/docs/cloud-integrations/semantic-layer/tableau) making query parsing more reliable. Some key fixes include: + - Error messages for unsupported joins between saved queries and ALL tables. + - Improved handling of queries when multiple tables are selected in a data source. + - Fixed a bug when an IN filter contained a lot of values. + - Better error messaging for queries that can't be parsed correctly. + ## October 2024 From c34e68e443bf544bd6b0fd1565eb9612d7ddd2fe Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Mon, 18 Nov 2024 15:24:53 -0500 Subject: [PATCH 10/81] update meta --- website/docs/docs/build/conversion-metrics.md | 12 +- website/docs/docs/build/dimensions.md | 107 +++++++++++++- website/docs/docs/build/entities.md | 69 ++++++++- website/docs/docs/build/measures.md | 136 ++++++++++++++++- website/docs/docs/build/simple.md | 4 +- .../docs/reference/resource-configs/meta.md | 138 ++++++++++++++---- website/snippets/_sl-measures-parameters.md | 1 + 7 files changed, 416 insertions(+), 51 deletions(-) diff --git a/website/docs/docs/build/conversion-metrics.md b/website/docs/docs/build/conversion-metrics.md index 2ef2c3910b9..979ee03e7b9 100644 --- a/website/docs/docs/build/conversion-metrics.md +++ b/website/docs/docs/build/conversion-metrics.md @@ -31,13 +31,13 @@ Note that we use the double colon (::) to indicate whether a parameter is nested | `entity` | The entity for each conversion event. | Required | | `calculation` | Method of calculation. Either `conversion_rate` or `conversions`. Defaults to `conversion_rate`. | Optional | | `base_measure` | A list of base measure inputs | Required | -| `base_measure:name` | The base conversion event measure. | Required | -| `base_measure:fill_nulls_with` | Set the value in your metric definition instead of null (such as zero). | Optional | -| `base_measure:join_to_timespine` | Boolean that indicates if the aggregated measure should be joined to the time spine table to fill in missing dates. Default `false`. | Optional | +| `base_measure::name` | The base conversion event measure. | Required | +| `base_measure::fill_nulls_with` | Set the value in your metric definition instead of null (such as zero). | Optional | +| `base_measure::join_to_timespine` | Boolean that indicates if the aggregated measure should be joined to the time spine table to fill in missing dates. Default `false`. | Optional | | `conversion_measure` | A list of conversion measure inputs. | Required | -| `conversion_measure:name` | The base conversion event measure.| Required | -| `conversion_measure:fill_nulls_with` | Set the value in your metric definition instead of null (such as zero). | Optional | -| `conversion_measure:join_to_timespine` | Boolean that indicates if the aggregated measure should be joined to the time spine table to fill in missing dates. Default `false`. | Optional | +| `conversion_measure::name` | The base conversion event measure.| Required | +| `conversion_measure::fill_nulls_with` | Set the value in your metric definition instead of null (such as zero). | Optional | +| `conversion_measure::join_to_timespine` | Boolean that indicates if the aggregated measure should be joined to the time spine table to fill in missing dates. Default `false`. | Optional | | `window` | The time window for the conversion event, such as 7 days, 1 week, 3 months. Defaults to infinity. | Optional | | `constant_properties` | List of constant properties. | Optional | | `base_property` | The property from the base semantic model that you want to hold constant. | Optional | diff --git a/website/docs/docs/build/dimensions.md b/website/docs/docs/build/dimensions.md index eb1809ea72b..424fd72e222 100644 --- a/website/docs/docs/build/dimensions.md +++ b/website/docs/docs/build/dimensions.md @@ -14,6 +14,26 @@ Groups are defined within semantic models, alongside entities and measures, and All dimensions require a `name`, `type`, and can optionally include an `expr` parameter. The `name` for your Dimension must be unique within the same semantic model. +:::tip +Note that we use the double colon (::) to indicate whether a parameter is nested within another parameter. So for example,` config::config_name` means the `config_name` parameter is nested under `config`. +::: + + + +| Parameter | Description | Type | +| --------- | ----------- | ---- | +| `name` | Refers to the name of the group that will be visible to the user in downstream tools. It can also serve as an alias if the column name or SQL query reference is different and provided in the `expr` parameter.

Dimension names should be unique within a semantic model, but they can be non-unique across different models as MetricFlow uses [joins](/docs/build/join-logic) to identify the right dimension. | Required | +| `type` | Specifies the type of group created in the semantic model. There are two types:

- **Categorical**: Describe attributes or features like geography or sales region.
- **Time**: Time-based dimensions like timestamps or dates. | Required | +| `type_params` | Specific type params such as if the time is primary or used as a partition | Required | +| `description` | A clear description of the dimension | Optional | +| `expr` | Defines the underlying column or SQL query for a dimension. If no `expr` is specified, MetricFlow will use the column with the same name as the group. You can use the column name itself to input a SQL expression. | Optional | +| `label` | A recommended string that defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | +| `config` | Use the [`config`](/reference/resource-properties/config) property to specify configurations for your metric. Supports [`meta`](/reference/resource-configs/meta), [`group`](/reference/resource-configs/group), and [`enabled`](/reference/resource-configs/enabled) configs. | Optional | + + + + + | Parameter | Description | Type | | --------- | ----------- | ---- | | `name` | Refers to the name of the group that will be visible to the user in downstream tools. It can also serve as an alias if the column name or SQL query reference is different and provided in the `expr` parameter.

Dimension names should be unique within a semantic model, but they can be non-unique across different models as MetricFlow uses [joins](/docs/build/join-logic) to identify the right dimension. | Required | @@ -22,7 +42,10 @@ All dimensions require a `name`, `type`, and can optionally include an `expr` pa | `description` | A clear description of the dimension | Optional | | `expr` | Defines the underlying column or SQL query for a dimension. If no `expr` is specified, MetricFlow will use the column with the same name as the group. You can use the column name itself to input a SQL expression. | Optional | | `label` | A recommended string that defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | -| `meta` | Use the [`meta` field](/reference/resource-configs/meta) to set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | +| `config` | Use the [`config`](/reference/resource-properties/config) property to specify configurations for your metric. Supports [`meta`](/reference/resource-configs/meta), [`group`](/reference/resource-configs/group), and [`enabled`](/reference/resource-configs/enabled) configs. | Optional | +| `config::meta` | Use [`meta`](/reference/resource-configs/meta), nested under `config`, to set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | + + Refer to the following for the complete specification for dimensions: @@ -38,6 +61,8 @@ dimensions: Refer to the following example to see how dimensions are used in a semantic model: + + ```yaml semantic_models: - name: transactions @@ -69,6 +94,40 @@ semantic_models: - name: type type: categorical ``` + + + + +```yaml +semantic_models: + - name: transactions + description: A record for every transaction that takes place. Carts are considered multiple transactions for each SKU. + model: {{ ref('fact_transactions') }} + defaults: + agg_time_dimension: order_date +# --- entities --- + entities: + - name: transaction + type: primary + ... +# --- measures --- + measures: + ... +# --- dimensions --- + dimensions: + - name: order_date + type: time + type_params: + time_granularity: day + label: "Date of transaction" # Recommend adding a label to provide more context to users consuming the data + expr: ts + - name: is_bulk + type: categorical + expr: case when quantity > 10 then true else false end + - name: type + type: categorical +``` + Dimensions are bound to the primary entity of the semantic model they are defined in. For example the dimension `type` is defined in a model that has `transaction` as a primary entity. `type` is scoped to the `transaction` entity, and to reference this dimension you would use the fully qualified dimension name i.e `transaction__type`. @@ -104,6 +163,8 @@ This section further explains the dimension definitions, along with examples. Di Categorical dimensions are used to group metrics by different attributes, features, or characteristics such as product type. They can refer to existing columns in your dbt model or be calculated using a SQL expression with the `expr` parameter. An example of a categorical dimension is `is_bulk_transaction`, which is a group created by applying a case statement to the underlying column `quantity`. This allows users to group or filter the data based on bulk transactions. + + ```yaml dimensions: - name: is_bulk_transaction @@ -112,6 +173,17 @@ dimensions: meta: usage: "Filter to identify bulk transactions, like where quantity > 10." ``` + + + + +```yaml +dimensions: + - name: is_bulk_transaction + type: categorical + expr: case when quantity > 10 then true else false end +``` + ## Time @@ -135,6 +207,8 @@ You can set `is_partition` for time to define specific time spans. Additionally, Use `is_partition: True` to show that a dimension exists over a specific time window. For example, a date-partitioned dimensional table. When you query metrics from different tables, the dbt Semantic Layer uses this parameter to ensure that the correct dimensional values are joined to measures. + + ```yaml dimensions: - name: created_at @@ -163,6 +237,37 @@ measures: expr: 1 agg: sum ``` + + + + +```yaml +dimensions: + - name: created_at + type: time + label: "Date of creation" + expr: ts_created # ts_created is the underlying column name from the table + is_partition: True + type_params: + time_granularity: day + - name: deleted_at + type: time + label: "Date of deletion" + expr: ts_deleted # ts_deleted is the underlying column name from the table + is_partition: True + type_params: + time_granularity: day + +measures: + - name: users_deleted + expr: 1 + agg: sum + agg_time_dimension: deleted_at + - name: users_created + expr: 1 + agg: sum +``` + diff --git a/website/docs/docs/build/entities.md b/website/docs/docs/build/entities.md index e4ed0773c3c..558dfd3aea4 100644 --- a/website/docs/docs/build/entities.md +++ b/website/docs/docs/build/entities.md @@ -95,17 +95,67 @@ Natural keys are columns or combinations of columns in a table that uniquely ide The following is the complete spec for entities: + + +```yaml +semantic_models: + - name: semantic_model_name + ..rest of the semantic model config + entities: + - name: entity_name ## Required + type: Primary, natural, foreign, or unique ## Required + description: A description of the field or role the entity takes in this table ## Optional + expr: The field that denotes that entity (transaction_id). ## Optional + Defaults to name if unspecified. + [config](/reference/resource-properties/config): Specify configurations for entity. ## Optional + [meta](/reference/resource-configs/meta): {} Set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes. ## Optional +``` + + + + +```yaml +semantic_models: + - name: semantic_model_name + ..rest of the semantic model config + entities: + - name: entity_name ## Required + type: Primary, or natural, or foreign, or unique ## Required + description: A description of the field or role the entity takes in this table ## Optional + expr: The field that denotes that entity (transaction_id). ## Optional + Defaults to name if unspecified. +``` + + +Here's an example of how to define entities in a semantic model: + + + ```yaml entities: - - name: transaction ## Required - type: Primary or natural or foreign or unique ## Required + - name: transaction + type: primary + expr: id_transaction + - name: order + type: foreign + expr: id_order + - name: user + type: foreign + expr: substring(id_order from 2) + entities: + - name: transaction + type: description: A description of the field or role the entity takes in this table ## Optional - expr: The field that denotes that entity (transaction_id). ## Optional + expr: The field that denotes that entity (transaction_id). Defaults to name if unspecified. + [config](/reference/resource-properties/config): + [meta](/reference/resource-configs/meta): + data_owner: "Finance team" ``` + + + -Here's an example of how to define entities in a semantic model: - ```yaml entities: - name: transaction @@ -117,11 +167,18 @@ entities: - name: user type: foreign expr: substring(id_order from 2) + entities: + - name: transaction + type: + description: A description of the field or role the entity takes in this table ## Optional + expr: The field that denotes that entity (transaction_id). + Defaults to name if unspecified. ``` + ## Combine columns with a key -If a table doesn't have any key (like a primary key), use _surrogate combination_ to form a key that will help you identify a record by combining two columns. This applies to any [entity type](/docs/build/entities#entity-types). For example, you can combine `date_key` and `brand_code` from the `raw_brand_target_weekly` table to form a _surrogate key_. The following example creates a surrogate key by joining `date_key` and `brand_code` using a pipe (`|`) as a separator. +If a table doesn't have any key (like a primary key), use _surrogate combination_ to form a key that will help you identify a record by combining two columns. This applies to any [entity type](/docs/build/entities#entity-types). For example, you can combine `date_key` and `brand_code` from the `raw_brand_target_weekly` table to form a _surrogate key_. The following example creates a surrogate key by joining `date_key` and `brand_code` using a pipe (`|`) as a separator. ```yaml diff --git a/website/docs/docs/build/measures.md b/website/docs/docs/build/measures.md index 977b630fada..a1470e72be9 100644 --- a/website/docs/docs/build/measures.md +++ b/website/docs/docs/build/measures.md @@ -10,6 +10,10 @@ Measures are aggregations performed on columns in your model. They can be used a Measures have several inputs, which are described in the following table along with their field types. +:::tip +Note that we use the double colon (::) to indicate whether a parameter is nested within another parameter. So for example,` config::config_name` means the `config_name` parameter is nested under `config`. +::: + import MeasuresParameters from '/snippets/_sl-measures-parameters.md'; @@ -18,16 +22,41 @@ import MeasuresParameters from '/snippets/_sl-measures-parameters.md'; An example of the complete YAML measures spec is below. The actual configuration of your measures will depend on the aggregation you're using. + + +```yaml +semantic_models: + - name: semantic_model_name + ..rest of the semantic model config + measures: + - name: The name of the measure + description: 'same as always' ## Optional + agg: the aggregation type. + expr: the field + agg_params: 'specific aggregation properties such as a percentile' ## Optional + agg_time_dimension: The time field. Defaults to the default agg time dimension for the semantic model. ## Optional + non_additive_dimension: 'Use these configs when you need non-additive dimensions.' ## Optional + [config](/reference/resource-properties/config): Use the config property to specify configurations for your measure. ## Optional + [meta](/reference/resource-configs/meta): {} Set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes. ## Optional +``` + + + + ```yaml -measures: - - name: The name of the measure - description: 'same as always' ## Optional - agg: the aggregation type. - expr: the field - agg_params: 'specific aggregation properties such as a percentile' ## Optional - agg_time_dimension: The time field. Defaults to the default agg time dimension for the semantic model. ## Optional - non_additive_dimension: 'Use these configs when you need non-additive dimensions.' ## Optional +semantic_models: + - name: semantic_model_name + ..rest of the semantic model config + measures: + - name: The name of the measure + description: 'same as always' ## Optional + agg: the aggregation type. + expr: the field + agg_params: 'specific aggregation properties such as a percentile' ## Optional + agg_time_dimension: The time field. Defaults to the default agg time dimension for the semantic model. ## Optional + non_additive_dimension: 'Use these configs when you need non-additive dimensions.' ## Optional ``` + ### Name @@ -96,6 +125,96 @@ If you use the `dayofweek` function in the `expr` parameter with the legacy Snow ### Model with different aggregations + + +```yaml +semantic_models: + - name: transactions + description: A record of every transaction that takes place. Carts are considered multiple transactions for each SKU. + model: ref('schema.transactions') + defaults: + agg_time_dimension: transaction_date + +# --- entities --- + entities: + - name: transaction_id + type: primary + - name: customer_id + type: foreign + - name: store_id + type: foreign + - name: product_id + type: foreign + +# --- measures --- + measures: + - name: transaction_amount_usd + description: Total USD value of transactions + expr: transaction_amount_usd + agg: sum + config: + meta: + used_in_reporting: true + - name: transaction_amount_usd_avg + description: Average USD value of transactions + expr: transaction_amount_usd + agg: average + - name: transaction_amount_usd_max + description: Maximum USD value of transactions + expr: transaction_amount_usd + agg: max + - name: transaction_amount_usd_min + description: Minimum USD value of transactions + expr: transaction_amount_usd + agg: min + - name: quick_buy_transactions + description: The total transactions bought as quick buy + expr: quick_buy_flag + agg: sum_boolean + - name: distinct_transactions_count + description: Distinct count of transactions + expr: transaction_id + agg: count_distinct + - name: transaction_amount_avg + description: The average value of transactions + expr: transaction_amount_usd + agg: average + - name: transactions_amount_usd_valid # Notice here how we use expr to compute the aggregation based on a condition + description: The total USD value of valid transactions only + expr: CASE WHEN is_valid = True then transaction_amount_usd else 0 end + agg: sum + - name: transactions + description: The average value of transactions. + expr: transaction_amount_usd + agg: average + - name: p99_transaction_value + description: The 99th percentile transaction value + expr: transaction_amount_usd + agg: percentile + agg_params: + percentile: .99 + use_discrete_percentile: False # False calculates the continuous percentile, True calculates the discrete percentile. + - name: median_transaction_value + description: The median transaction value + expr: transaction_amount_usd + agg: median + +# --- dimensions --- + dimensions: + - name: transaction_date + type: time + expr: date_trunc('day', ts) # expr refers to underlying column ts + type_params: + time_granularity: day + - name: is_bulk_transaction + type: categorical + expr: case when quantity > 10 then true else false end + +``` + + + + ```yaml semantic_models: - name: transactions @@ -177,6 +296,7 @@ semantic_models: expr: case when quantity > 10 then true else false end ``` + ### Non-additive dimensions diff --git a/website/docs/docs/build/simple.md b/website/docs/docs/build/simple.md index f57d498d290..4a3aac32708 100644 --- a/website/docs/docs/build/simple.md +++ b/website/docs/docs/build/simple.md @@ -24,8 +24,8 @@ Note that we use the double colon (::) to indicate whether a parameter is nested | `type_params` | The type parameters of the metric. | Required | | `measure` | A list of measure inputs | Required | | `measure:name` | The measure you're referencing. | Required | -| `measure:fill_nulls_with` | Set the value in your metric definition instead of null (such as zero). | Optional | -| `measure:join_to_timespine` | Boolean that indicates if the aggregated measure should be joined to the time spine table to fill in missing dates. Default `false`. | Optional | +| `measure::fill_nulls_with` | Set the value in your metric definition instead of null (such as zero). | Optional | +| `measure::join_to_timespine` | Boolean that indicates if the aggregated measure should be joined to the time spine table to fill in missing dates. Default `false`. | Optional | The following displays the complete specification for simple metrics, along with an example. diff --git a/website/docs/reference/resource-configs/meta.md b/website/docs/reference/resource-configs/meta.md index fa0d48c299e..3d861fc5360 100644 --- a/website/docs/reference/resource-configs/meta.md +++ b/website/docs/reference/resource-configs/meta.md @@ -16,10 +16,9 @@ hide_table_of_contents: true { label: 'Analyses', value: 'analyses', }, { label: 'Macros', value: 'macros', }, { label: 'Exposures', value: 'exposures', }, - { label: 'Semantic Models', value: 'semantic models', }, + { label: 'Semantic models', value: 'semantic models', }, { label: 'Metrics', value: 'metrics', }, { label: 'Saved queries', value: 'saved queries', }, - { label: 'Dimensions', value: 'dimensions', }, ] }> @@ -180,6 +179,10 @@ exposures: +Configure `meta` in the your semantic models YAML file or under the `semantic-models` config block in the `dbt_project.yml` file. + + + ```yml @@ -191,6 +194,39 @@ semantic_models: ``` + + + + +[Dimensions](/docs/build/dimensions), [entities](/docs/build/entities), and [measures](/docs/build/measures) can also have their own `meta` configurations. + + + +```yml +semantic_models: + - name: semantic_model_name + config: + meta: {} + + dimensions: + - name: dimension_name + config: + meta: {} + + entities: + - name: entity_name + config: + meta: {} + + measures: + - name: measure_name + config: + meta: {} + +``` + + + The `meta` config can also be defined under the `semantic-models` config block in `dbt_project.yml`. See [configs and properties](/reference/configs-and-properties) for details. @@ -250,25 +286,7 @@ saved_queries: ``` - - - - - -```yml -semantic_models: - - name: semantic_model - ... - [dimensions](/docs/build/dimensions): - - name: order_date - config: - meta: {} -``` - - - - ## Definition @@ -360,9 +378,54 @@ models: -### Assign owner and additional metadata to dimensions +### Assign meta to semantic model + -The following example shows how to assign a `data_owner` and additional metadata value to a dimension in a semantic model. +The following example shows how to assign a `meta` value to a semantic model in the `semantic_model.yml` file and `dbt_project.yml` file: + + + + +```yaml +semantic_models: + - name: transaction + model: ref('fact_transactions') + description: "Transaction fact table at the transaction level. This table contains one row per transaction and includes the transaction timestamp." + defaults: + agg_time_dimension: transaction_date + config: + meta: + data_owner: "Finance team" + used_in_reporting: true +``` + + + + + +```yaml +semantic-models: + jaffle_shop: + +meta: + used_in_reporting: true +``` + + + +### Assign meta to dimensions, measures, entities + + + +Available in dbt version 1.9 and later. + + + + + + + + +The following example shows how to assign a `meta` value to a [dimension](/docs/build/dimensions), [entity](/docs/build/entities), and [measure](/docs/build/measures) in a semantic model: @@ -370,16 +433,32 @@ The following example shows how to assign a `data_owner` and additional metadata semantic_models: - name: semantic_model ... - [dimensions](/docs/build/dimensions): - - name: order_date - meta: - data_owner: "Finance team" - used_in_reporting: true + dimensions: + - name: order_date + type: time + meta: + data_owner: "Finance team" + used_in_reporting: true + entities: + - name: customer_id + type: primary + meta: + description: "Unique identifier for customers" + data_owner: "Sales team" + used_in_reporting: false + measures: + - name: count_of_users + expr: user_id + meta: + used_in_reporting: true ``` + + + -This second example shows how to assign a `data_owner` and additional metadata value to a dimension in the `dbt_project.yml` file using the `+meta` syntax. +This second example shows how to assign a `data_owner` and additional metadata value to a dimension in the `dbt_project.yml` file using the `+meta` syntax. The similar syntax can be used for entities and measures. @@ -395,3 +474,6 @@ semantic-models: ``` + + + diff --git a/website/snippets/_sl-measures-parameters.md b/website/snippets/_sl-measures-parameters.md index 728d63c6b4f..09916714b96 100644 --- a/website/snippets/_sl-measures-parameters.md +++ b/website/snippets/_sl-measures-parameters.md @@ -9,3 +9,4 @@ | `agg_time_dimension` | The time field. Defaults to the default agg time dimension for the semantic model. | Optional | 1.6 and higher | | `label` | String that defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as orders_total or "orders_total"). Available in dbt version 1.7 or higher. | Optional | `create_metric` | Create a `simple` metric from a measure by setting `create_metric: True`. The `label` and `description` attributes will be automatically propagated to the created metric. Available in dbt version 1.7 or higher. | Optional | +| `config::meta` | Use [`meta`](/reference/resource-configs/meta), nested under [`config`](/reference/resource-properties/config), to set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | From 7824b08ee66b24685b0caa9550fdf17ebf6bff83 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Mon, 18 Nov 2024 16:32:51 -0500 Subject: [PATCH 11/81] update --- website/docs/docs/build/dimensions.md | 9 ++------- website/docs/docs/build/measures.md | 4 ---- website/snippets/_sl-measures-parameters.md | 2 +- 3 files changed, 3 insertions(+), 12 deletions(-) diff --git a/website/docs/docs/build/dimensions.md b/website/docs/docs/build/dimensions.md index 424fd72e222..a53869bef3c 100644 --- a/website/docs/docs/build/dimensions.md +++ b/website/docs/docs/build/dimensions.md @@ -14,10 +14,6 @@ Groups are defined within semantic models, alongside entities and measures, and All dimensions require a `name`, `type`, and can optionally include an `expr` parameter. The `name` for your Dimension must be unique within the same semantic model. -:::tip -Note that we use the double colon (::) to indicate whether a parameter is nested within another parameter. So for example,` config::config_name` means the `config_name` parameter is nested under `config`. -::: - | Parameter | Description | Type | @@ -28,7 +24,7 @@ Note that we use the double colon (::) to indicate whether a parameter is nested | `description` | A clear description of the dimension | Optional | | `expr` | Defines the underlying column or SQL query for a dimension. If no `expr` is specified, MetricFlow will use the column with the same name as the group. You can use the column name itself to input a SQL expression. | Optional | | `label` | A recommended string that defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | -| `config` | Use the [`config`](/reference/resource-properties/config) property to specify configurations for your metric. Supports [`meta`](/reference/resource-configs/meta), [`group`](/reference/resource-configs/group), and [`enabled`](/reference/resource-configs/enabled) configs. | Optional | +| `config` | Use the [`config`](/reference/resource-properties/config) property to specify configurations for your metric. Supports the [`meta`](/reference/resource-configs/meta) property, nested under `config`. | Optional | @@ -42,8 +38,7 @@ Note that we use the double colon (::) to indicate whether a parameter is nested | `description` | A clear description of the dimension | Optional | | `expr` | Defines the underlying column or SQL query for a dimension. If no `expr` is specified, MetricFlow will use the column with the same name as the group. You can use the column name itself to input a SQL expression. | Optional | | `label` | A recommended string that defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | -| `config` | Use the [`config`](/reference/resource-properties/config) property to specify configurations for your metric. Supports [`meta`](/reference/resource-configs/meta), [`group`](/reference/resource-configs/group), and [`enabled`](/reference/resource-configs/enabled) configs. | Optional | -| `config::meta` | Use [`meta`](/reference/resource-configs/meta), nested under `config`, to set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | +| `config` | Use the [`config`](/reference/resource-properties/config) property to specify configurations for your metric. Supports the [`meta`](/reference/resource-configs/meta) configs. | Optional | diff --git a/website/docs/docs/build/measures.md b/website/docs/docs/build/measures.md index a1470e72be9..c12cf67fdc3 100644 --- a/website/docs/docs/build/measures.md +++ b/website/docs/docs/build/measures.md @@ -10,10 +10,6 @@ Measures are aggregations performed on columns in your model. They can be used a Measures have several inputs, which are described in the following table along with their field types. -:::tip -Note that we use the double colon (::) to indicate whether a parameter is nested within another parameter. So for example,` config::config_name` means the `config_name` parameter is nested under `config`. -::: - import MeasuresParameters from '/snippets/_sl-measures-parameters.md'; diff --git a/website/snippets/_sl-measures-parameters.md b/website/snippets/_sl-measures-parameters.md index 09916714b96..8058474d1ef 100644 --- a/website/snippets/_sl-measures-parameters.md +++ b/website/snippets/_sl-measures-parameters.md @@ -9,4 +9,4 @@ | `agg_time_dimension` | The time field. Defaults to the default agg time dimension for the semantic model. | Optional | 1.6 and higher | | `label` | String that defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as orders_total or "orders_total"). Available in dbt version 1.7 or higher. | Optional | `create_metric` | Create a `simple` metric from a measure by setting `create_metric: True`. The `label` and `description` attributes will be automatically propagated to the created metric. Available in dbt version 1.7 or higher. | Optional | -| `config::meta` | Use [`meta`](/reference/resource-configs/meta), nested under [`config`](/reference/resource-properties/config), to set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | +| `config` | Use the [`config`](/reference/resource-properties/config) property to specify configurations for your metric. Supports the [`meta`](/reference/resource-configs/meta) property, nested under `config`. | Optional | From 16faf8cde0cd3659f8d300a81214f7c7be61f7dd Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 10 Dec 2024 10:47:54 +0000 Subject: [PATCH 12/81] Update exports.md --- website/docs/docs/use-dbt-semantic-layer/exports.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/website/docs/docs/use-dbt-semantic-layer/exports.md b/website/docs/docs/use-dbt-semantic-layer/exports.md index d5285c8f779..2ecffc508a2 100644 --- a/website/docs/docs/use-dbt-semantic-layer/exports.md +++ b/website/docs/docs/use-dbt-semantic-layer/exports.md @@ -24,10 +24,7 @@ Essentially, exports are like any other table in your data platform — they ## Benefits of exports -The following section explains the main benefits of using exports, including: -- [DRY representation](#dry-representation) -- [Easier changes](#easier-changes) -- [Caching](#caching) +The following section explains the main benefits of using exports: From 7118aa5b95586e7c6beacc67617868aa867e2314 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 10 Dec 2024 10:54:15 +0000 Subject: [PATCH 13/81] Update dimensions.md --- website/docs/docs/build/dimensions.md | 1 + 1 file changed, 1 insertion(+) diff --git a/website/docs/docs/build/dimensions.md b/website/docs/docs/build/dimensions.md index 70715537f9a..f1a2f10bf89 100644 --- a/website/docs/docs/build/dimensions.md +++ b/website/docs/docs/build/dimensions.md @@ -22,6 +22,7 @@ All dimensions require a `name`, `type`, and can optionally include an `expr` pa | `description` | A clear description of the dimension. | Optional | String | | `expr` | Defines the underlying column or SQL query for a dimension. If no `expr` is specified, MetricFlow will use the column with the same name as the group. You can use the column name itself to input a SQL expression. | Optional | String | | `label` | Defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | String | +| [`meta`](/reference/resource-configs/meta) | Set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes. | Optional | {} | Refer to the following for the complete specification for dimensions: From e7f407f013642f639012c34a7560a6941c6d0906 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 10 Dec 2024 10:59:47 +0000 Subject: [PATCH 14/81] Update website/docs/docs/build/dimensions.md --- website/docs/docs/build/dimensions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/dimensions.md b/website/docs/docs/build/dimensions.md index f1a2f10bf89..609b2168906 100644 --- a/website/docs/docs/build/dimensions.md +++ b/website/docs/docs/build/dimensions.md @@ -22,7 +22,7 @@ All dimensions require a `name`, `type`, and can optionally include an `expr` pa | `description` | A clear description of the dimension. | Optional | String | | `expr` | Defines the underlying column or SQL query for a dimension. If no `expr` is specified, MetricFlow will use the column with the same name as the group. You can use the column name itself to input a SQL expression. | Optional | String | | `label` | Defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | String | -| [`meta`](/reference/resource-configs/meta) | Set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes. | Optional | {} | +| [`meta`](/reference/resource-configs/meta) | Set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes. | Optional | | Refer to the following for the complete specification for dimensions: From 5617cf82b9666c01087d5a0c7d3afa1f63d60ded Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 10 Dec 2024 11:18:14 +0000 Subject: [PATCH 15/81] Update dimensions.md --- website/docs/docs/build/dimensions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/dimensions.md b/website/docs/docs/build/dimensions.md index 609b2168906..4ad06e5626e 100644 --- a/website/docs/docs/build/dimensions.md +++ b/website/docs/docs/build/dimensions.md @@ -22,7 +22,7 @@ All dimensions require a `name`, `type`, and can optionally include an `expr` pa | `description` | A clear description of the dimension. | Optional | String | | `expr` | Defines the underlying column or SQL query for a dimension. If no `expr` is specified, MetricFlow will use the column with the same name as the group. You can use the column name itself to input a SQL expression. | Optional | String | | `label` | Defines the display value in downstream tools. Accepts plain text, spaces, and quotes (such as `orders_total` or `"orders_total"`). | Optional | String | -| [`meta`](/reference/resource-configs/meta) | Set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes. | Optional | | +| [`meta`](/reference/resource-configs/meta) | Set metadata for a resource and organize resources. Accepts plain text, spaces, and quotes. | Optional | Dictionary | Refer to the following for the complete specification for dimensions: From e2ccfb34d341c3bf3e8262ad02062a2e2164626d Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Wed, 11 Dec 2024 11:27:31 +0000 Subject: [PATCH 16/81] update to saved queries --- website/docs/docs/build/saved-queries.md | 14 ++-- .../docs/docs/dbt-versions/release-notes.md | 12 +-- .../docs/reference/resource-configs/tags.md | 77 ++++++++++--------- 3 files changed, 54 insertions(+), 49 deletions(-) diff --git a/website/docs/docs/build/saved-queries.md b/website/docs/docs/build/saved-queries.md index 68876b65160..840b1ebb95c 100644 --- a/website/docs/docs/build/saved-queries.md +++ b/website/docs/docs/build/saved-queries.md @@ -26,7 +26,7 @@ Note that we use the double colon (::) to indicate whether a parameter is nested | `name` | String | Required | Name of the saved query object. | | `description` | String | Required | A description of the saved query. | | `label` | String | Required | The display name for your saved query. This value will be shown in downstream tools. | -| `config` | String | Optional | Use the [`config`](/reference/resource-properties/config) property to specify configurations for your saved query. Supports `cache`, [`enabled`](/reference/resource-configs/enabled), `export_as`, [`group`](/reference/resource-configs/group), [`meta`](/reference/resource-configs/meta), and [`schema`](/reference/resource-configs/schema) configurations. | +| `config` | String | Optional | Use the [`config`](/reference/resource-properties/config) property to specify configurations for your saved query. Supports `cache`, [`enabled`](/reference/resource-configs/enabled), `export_as`, [`group`](/reference/resource-configs/group), [`meta`](/reference/resource-configs/meta), [`tags`](/reference/resource-configs/tags), and [`schema`](/reference/resource-configs/schema) configurations. | | `config::cache::enabled` | Object | Optional | An object with a sub-key used to specify if a saved query should populate the [cache](/docs/use-dbt-semantic-layer/sl-cache). Accepts sub-key `true` or `false`. Defaults to `false` | | `query_params` | Structure | Required | Contains the query parameters. | | `query_params::metrics` | List or String | Optional | A list of the metrics to be used in the query as specified in the command line interface. | @@ -38,7 +38,6 @@ Note that we use the double colon (::) to indicate whether a parameter is nested | `exports::config::export_as` | String | Required | The type of export to run. Options include table or view currently and cache in the near future. | | `exports::config::schema` | String | Optional | The [schema](/reference/resource-configs/schema) for creating the table or view. This option cannot be used for caching. | | `exports::config::alias` | String | Optional | The table [alias](/reference/resource-configs/alias) used to write to the table or view. This option cannot be used for caching. | -| `exports::config::tags` | String | Optional | Apply a [tag](/reference/resource-configs/tags) (or list of tags) to a resource. Tags help organize and filter resources in dbt. Nested within `exports` and `config`. | @@ -108,7 +107,8 @@ saved_queries: label: Test saved query config: cache: - enabled: true # Or false if you want it disabled by default + [enabled](/reference/resource-configs/enabled): true | false + [tags](/reference/resource-configs/tags): 'my_tag' query_params: metrics: - simple_metric @@ -123,7 +123,6 @@ saved_queries: export_as: table alias: my_export_alias schema: my_export_schema_name - tags: 'my_tag' ``` @@ -243,6 +242,9 @@ The following is an example of a saved query with an export: saved_queries: - name: order_metrics description: Relevant order metrics + config: + tags: + - order_metrics query_params: metrics: - orders @@ -262,9 +264,7 @@ saved_queries: config: export_as: table # Options available: table, view [alias](/reference/resource-configs/alias): my_export_alias # Optional - defaults to Export name - [schema](/reference/resource-configs/schema): my_export_schema_name # Optional - defaults to deployment schema - [tags](/reference/resource-configs/tags): 'my_tag' - + [schema](/reference/resource-configs/schema): my_export_schema_name # Optional - defaults to deployment schema ``` diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index ad8e2d3a9a9..88d3d2b88f2 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -20,14 +20,16 @@ Release notes are grouped by month for both multi-tenant and virtual private clo ## December 2024 -- **New**: Exports now support [tags](/reference/resource-configs/tags) in dbt. Tags allow you to categorize your resources and filter them. You can add tags to your [exports](/docs/build/saved-queries#configure-exports) in the `semantic_model.yml` file or `dbt_project.yml` file. For example: +- **New**: Saved queries now support [tags](/reference/resource-configs/tags), which allow you to categorize your resources and filter them. Add tags to your [saved queries](/docs/build/saved-queries) in the `semantic_model.yml` file or `dbt_project.yml` file. For example: + ```yml - exports: - - name: export_name - tags: ['export_tag'] - ... + [saved-queries](/docs/build/saved-queries): + jaffle_shop: + customer_order_metrics: + +tags: order_metrics ``` + - **New**: [Model notifications](/docs/deploy/model-notifications) are now generally available in dbt Cloud. These notifications alert model owners through email about any issues encountered by models and tests as soon as they occur while running a job. - **New**: You can now use your [Azure OpenAI key](/docs/cloud/account-integrations?ai-integration=azure#ai-integrations) (available in beta) to use dbt Cloud features like [dbt Copilot](/docs/cloud/dbt-copilot) and [Ask dbt](/docs/cloud-integrations/snowflake-native-app) . Additionally, you can use your own [OpenAI API key](/docs/cloud/account-integrations?ai-integration=openai#ai-integrations) or use [dbt Labs-managed OpenAI](/docs/cloud/account-integrations?ai-integration=dbtlabs#ai-integrations) key. Refer to [AI integrations](/docs/cloud/account-integrations#ai-integrations) for more information. - **New**: The [`hard_deletes`](/reference/resource-configs/hard-deletes) config gives you more control on how to handle deleted rows from the source. Supported options are `ignore` (default), `invalidate` (replaces the legacy `invalidate_hard_deletes=true`), and `new_record`. Note that `new_record` will create a new metadata column in the snapshot table. diff --git a/website/docs/reference/resource-configs/tags.md b/website/docs/reference/resource-configs/tags.md index e8838a72422..45c4e719059 100644 --- a/website/docs/reference/resource-configs/tags.md +++ b/website/docs/reference/resource-configs/tags.md @@ -20,15 +20,15 @@ datatype: string | [string] ```yml -models: +[models](/reference/model-configs): [](/reference/resource-configs/resource-path): +tags: | [] -snapshots: +[snapshots](/reference/snapshot-configs): [](/reference/resource-configs/resource-path): +tags: | [] -seeds: +[seeds](/reference/seed-configs): [](/reference/resource-configs/resource-path): +tags: | [] @@ -39,19 +39,19 @@ seeds: ```yml -models: +[models](/reference/model-configs): [](/reference/resource-configs/resource-path): +tags: | [] -snapshots: +[snapshots](/reference/snapshot-configs): [](/reference/resource-configs/resource-path): +tags: | [] -seeds: +[seeds](/reference/seed-configs): [](/reference/resource-configs/resource-path): +tags: | [] -exports: +[saved-queries:](/docs/build/saved-queries) [](/reference/resource-configs/resource-path): +tags: | [] @@ -64,38 +64,45 @@ exports: - + -```yml -version: 2 +The following examples show how to add tags to dbt resources in YAML files. Replace `resource_type` with `models`, `snapshots`, `seeds`, or `saved_queries` as appropriate. + -models: - - name: model_name - config: - tags: | [] + + +The following examples show how to add tags to dbt resources in YAML files. Replace `resource_type` with `models`, `snapshots`, or `seeds` as appropriate. + + + +```yaml +resource_type: + - name: resource_name + config: + tags: string + # Optional: Add specific properties for models columns: - name: column_name - tags: [] + tags: string tests: - : + test-name: config: - tags: | [] + tags: string ``` - -```jinja - + +```sql {{ config( tags="" | [""] ) }} - ``` + @@ -191,22 +198,22 @@ seeds: -### Apply tags to exports +### Apply tags to saved queries -Applying tags to exports is only available in dbt Core versions 1.9 and later. +Applying tags to saved queries is only available in dbt Core versions 1.9 and later. -This following example shows how to apply tags to an export in the `dbt_project.yml` file. The export is then tagged with `order_metrics`. +This following example shows how to apply a tag to a saved query in the `dbt_project.yml` file. The saved query is then tagged with `order_metrics`. ```yml -[exports](/docs/build/saved-queries#configure-exports): +[saved-queries](/docs/build/saved-queries): jaffle_shop: customer_order_metrics: +tags: order_metrics @@ -214,23 +221,19 @@ This following example shows how to apply tags to an export in the `dbt_project. -The second example shows how to apply tags to an export in the `semantic_model.yml` file. The export is then tagged with `order_metrics` and `hourly`. +The second example shows how to apply multiple tags to a saved query in the `semantic_model.yml` file. The saved query is then tagged with `order_metrics` and `hourly`. ```yaml saved_queries: - - name: order_metrics - ... - exports: - - name: hourly_order_metrics - config: - alias: my_export_alias - export_as: table - schema: my_export_schema_name - tags: - - order_metrics - - hourly + - name: test_saved_query + description: "{{ doc('saved_query_description') }}" + label: Test saved query + config: + tags: + - order_metrics + - hourly ``` From af4165628afc27b9323cf788d9e059ba0abdccc9 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Fri, 13 Dec 2024 11:39:50 +0000 Subject: [PATCH 17/81] add config --- website/docs/docs/build/dimensions.md | 15 ++++++---- .../docs/reference/resource-configs/meta.md | 29 +++++++++++-------- 2 files changed, 26 insertions(+), 18 deletions(-) diff --git a/website/docs/docs/build/dimensions.md b/website/docs/docs/build/dimensions.md index 4ad06e5626e..25a1c729a7a 100644 --- a/website/docs/docs/build/dimensions.md +++ b/website/docs/docs/build/dimensions.md @@ -62,8 +62,9 @@ semantic_models: type_params: time_granularity: day label: "Date of transaction" # Recommend adding a label to provide more context to users consuming the data - meta: - data_owner: "Finance team" + config: + meta: + data_owner: "Finance team" expr: ts - name: is_bulk type: categorical @@ -147,8 +148,9 @@ dimensions: - name: is_bulk_transaction type: categorical expr: case when quantity > 10 then true else false end - meta: - usage: "Filter to identify bulk transactions, like where quantity > 10." + config: + meta: + usage: "Filter to identify bulk transactions, like where quantity > 10." ``` @@ -192,8 +194,9 @@ dimensions: type: time label: "Date of creation" expr: ts_created # ts_created is the underlying column name from the table - meta: - notes: "Only valid for orders from 2022 onward" + config: + meta: + notes: "Only valid for orders from 2022 onward" is_partition: True type_params: time_granularity: day diff --git a/website/docs/reference/resource-configs/meta.md b/website/docs/reference/resource-configs/meta.md index dcf04354dd6..7214dac560b 100644 --- a/website/docs/reference/resource-configs/meta.md +++ b/website/docs/reference/resource-configs/meta.md @@ -436,21 +436,24 @@ semantic_models: dimensions: - name: order_date type: time - meta: - data_owner: "Finance team" - used_in_reporting: true + config: + meta: + data_owner: "Finance team" + used_in_reporting: true entities: - name: customer_id type: primary - meta: - description: "Unique identifier for customers" - data_owner: "Sales team" - used_in_reporting: false + config: + meta: + description: "Unique identifier for customers" + data_owner: "Sales team" + used_in_reporting: false measures: - name: count_of_users expr: user_id - meta: - used_in_reporting: true + config: + meta: + used_in_reporting: true ``` @@ -468,11 +471,13 @@ semantic-models: ... [dimensions](/docs/build/dimensions): - name: order_date - +meta: - data_owner: "Finance team" - used_in_reporting: true + config: + meta: + data_owner: "Finance team" + used_in_reporting: true ``` + From cca64d35b1ec3cf592f84e3d77bcd7e6bcd07469 Mon Sep 17 00:00:00 2001 From: Ben Cassell <98852248+benc-db@users.noreply.github.com> Date: Tue, 17 Dec 2024 09:23:31 -0800 Subject: [PATCH 18/81] Update databricks-configs.md for merge settings --- website/docs/reference/resource-configs/databricks-configs.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/reference/resource-configs/databricks-configs.md b/website/docs/reference/resource-configs/databricks-configs.md index 6ac3e23c113..53aa0fe4375 100644 --- a/website/docs/reference/resource-configs/databricks-configs.md +++ b/website/docs/reference/resource-configs/databricks-configs.md @@ -437,8 +437,8 @@ Beginning with 1.9, `merge` behavior can be modified with the following addition - `skip_not_matched_step`: If set to `true`, the 'not matched' clause will not be included. - `matched_condition`: Condition to apply to the `WHEN MATCHED` clause. You should use the `target_alias` and `source_alias` to write a conditional expression, such as `DBT_INTERNAL_DEST.col1 = hash(DBT_INTERNAL_SOURCE.col2, DBT_INTERNAL_SOURCE.col3)`. This condition further restricts the matched set of rows. - `not_matched_condition`: Condition to apply to the `WHEN NOT MATCHED [BY TARGET]` clause. This condition further restricts the set of rows in the target that do not match the source that will be inserted into the merged table. -- `not_matched_by_source_condition`: Condition to apply to the further filter `WHEN NOT MATCHED BY SOURCE` clause. Only used in conjunction with `not_matched_by_source_action: delete`. -- `not_matched_by_source_action`: If set to `delete`, a `DELETE` clause is added to the merge statement for `WHEN NOT MATCHED BY SOURCE`. +- `not_matched_by_source_condition`: Condition to apply to the further filter `WHEN NOT MATCHED BY SOURCE` clause. Only used in conjunction with `not_matched_by_source_action`. +- `not_matched_by_source_action`: Action to apply when the condition is met. For example: `update set t.attr1 = 'deleted', t.tech_change_ts = current_timestamp()`. - `merge_with_schema_evolution`: If set to `true`, the merge statement includes the `WITH SCHEMA EVOLUTION` clause. For more details on the meaning of each merge clause, please see [the Databricks documentation](https://docs.databricks.com/en/sql/language-manual/delta-merge-into.html). From 8f4bdbd6e78c3cbc235de8c78a7490e9bad5d006 Mon Sep 17 00:00:00 2001 From: Ben Cassell <98852248+benc-db@users.noreply.github.com> Date: Tue, 17 Dec 2024 10:03:39 -0800 Subject: [PATCH 19/81] Update databricks-configs.md --- website/docs/reference/resource-configs/databricks-configs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-configs/databricks-configs.md b/website/docs/reference/resource-configs/databricks-configs.md index 53aa0fe4375..d7b5a79db37 100644 --- a/website/docs/reference/resource-configs/databricks-configs.md +++ b/website/docs/reference/resource-configs/databricks-configs.md @@ -438,7 +438,7 @@ Beginning with 1.9, `merge` behavior can be modified with the following addition - `matched_condition`: Condition to apply to the `WHEN MATCHED` clause. You should use the `target_alias` and `source_alias` to write a conditional expression, such as `DBT_INTERNAL_DEST.col1 = hash(DBT_INTERNAL_SOURCE.col2, DBT_INTERNAL_SOURCE.col3)`. This condition further restricts the matched set of rows. - `not_matched_condition`: Condition to apply to the `WHEN NOT MATCHED [BY TARGET]` clause. This condition further restricts the set of rows in the target that do not match the source that will be inserted into the merged table. - `not_matched_by_source_condition`: Condition to apply to the further filter `WHEN NOT MATCHED BY SOURCE` clause. Only used in conjunction with `not_matched_by_source_action`. -- `not_matched_by_source_action`: Action to apply when the condition is met. For example: `update set t.attr1 = 'deleted', t.tech_change_ts = current_timestamp()`. +- `not_matched_by_source_action`: Action to apply when the condition is met. For example: `not_matched_by_source_action: "update set t.attr1 = 'deleted', t.tech_change_ts = current_timestamp()"`. - `merge_with_schema_evolution`: If set to `true`, the merge statement includes the `WITH SCHEMA EVOLUTION` clause. For more details on the meaning of each merge clause, please see [the Databricks documentation](https://docs.databricks.com/en/sql/language-manual/delta-merge-into.html). From ce615e228ad7494e70ced88f5e2c18aabda70269 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Wed, 18 Dec 2024 13:34:56 -0500 Subject: [PATCH 20/81] Update website/docs/reference/resource-configs/databricks-configs.md --- website/docs/reference/resource-configs/databricks-configs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-configs/databricks-configs.md b/website/docs/reference/resource-configs/databricks-configs.md index d7b5a79db37..1ee89efc95c 100644 --- a/website/docs/reference/resource-configs/databricks-configs.md +++ b/website/docs/reference/resource-configs/databricks-configs.md @@ -438,7 +438,7 @@ Beginning with 1.9, `merge` behavior can be modified with the following addition - `matched_condition`: Condition to apply to the `WHEN MATCHED` clause. You should use the `target_alias` and `source_alias` to write a conditional expression, such as `DBT_INTERNAL_DEST.col1 = hash(DBT_INTERNAL_SOURCE.col2, DBT_INTERNAL_SOURCE.col3)`. This condition further restricts the matched set of rows. - `not_matched_condition`: Condition to apply to the `WHEN NOT MATCHED [BY TARGET]` clause. This condition further restricts the set of rows in the target that do not match the source that will be inserted into the merged table. - `not_matched_by_source_condition`: Condition to apply to the further filter `WHEN NOT MATCHED BY SOURCE` clause. Only used in conjunction with `not_matched_by_source_action`. -- `not_matched_by_source_action`: Action to apply when the condition is met. For example: `not_matched_by_source_action: "update set t.attr1 = 'deleted', t.tech_change_ts = current_timestamp()"`. +- `not_matched_by_source_action`: The action to apply when the condition is met. Configure as an expression. For example: `not_matched_by_source_action: "update set t.attr1 = 'deleted', t.tech_change_ts = current_timestamp()"`. - `merge_with_schema_evolution`: If set to `true`, the merge statement includes the `WITH SCHEMA EVOLUTION` clause. For more details on the meaning of each merge clause, please see [the Databricks documentation](https://docs.databricks.com/en/sql/language-manual/delta-merge-into.html). From 72bd8d8c836c7eff4bba48f62edd56e713054a72 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Thu, 19 Dec 2024 13:27:33 +0000 Subject: [PATCH 21/81] centarlize content --- .../cloud/dbt-cloud-ide/ide-user-interface.md | 2 +- .../git/git-configuration-in-dbt-cloud.md | 37 ++++++++++++------- .../cloud/git/import-a-project-by-git-url.md | 8 ++-- .../git/managed-repository.md | 5 ++- .../docs/docs/collaborate/git/pr-template.md | 2 +- website/docs/faqs/Git/gitignore.md | 11 ++++-- .../ide-session-unknown-error.md | 2 +- website/docs/guides/dbt-python-snowpark.md | 4 +- website/docs/guides/sl-snowflake-qs.md | 4 +- website/docs/guides/snowflake-qs.md | 2 +- website/sidebars.js | 2 +- website/snippets/available-git-providers.md | 2 +- website/vercel.json | 7 +++- 13 files changed, 54 insertions(+), 34 deletions(-) rename website/docs/docs/{collaborate => cloud}/git/managed-repository.md (83%) diff --git a/website/docs/docs/cloud/dbt-cloud-ide/ide-user-interface.md b/website/docs/docs/cloud/dbt-cloud-ide/ide-user-interface.md index 36c6cc898dc..72c1fbe7af6 100644 --- a/website/docs/docs/cloud/dbt-cloud-ide/ide-user-interface.md +++ b/website/docs/docs/cloud/dbt-cloud-ide/ide-user-interface.md @@ -64,7 +64,7 @@ The IDE features some delightful tools and layouts to make it easier for you to - - Use the **Prune branches** option to remove local branches that have already been deleted from the remote repository. Selecting this triggers a [pop-up modal](#prune-branches-modal), where you can confirm the deletion of the specific local branches, keeping your branch management tidy. Note that this won't delete the branch you're currently on. Pruning branches isn't available for [managed repositories](/docs/collaborate/git/managed-repository) because they don't have a typical remote setup, which prevents remote branch deletion. + - Use the **Prune branches** option to remove local branches that have already been deleted from the remote repository. Selecting this triggers a [pop-up modal](#prune-branches-modal), where you can confirm the deletion of the specific local branches, keeping your branch management tidy. Note that this won't delete the branch you're currently on. Pruning branches isn't available for [managed repositories](/docs/cloud/git/managed-repository) because they don't have a typical remote setup, which prevents remote branch deletion. ## Additional editing features diff --git a/website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md b/website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md index fb8c0186236..fabd2073995 100644 --- a/website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md +++ b/website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md @@ -1,37 +1,46 @@ --- -title: "Git configuration in dbt Cloud" +title: "Configure Git in dbt Cloud" description: "Learn about the Git providers supported in dbt Cloud" -pagination_next: "docs/cloud/git/import-a-project-by-git-url" -pagination_prev: null +hide_table_of_contents: true +pagination_next: "docs/cloud/git/managed-repository" --- -
+[Version control](/docs/collaborate/git/version-control-basics) is an essential part of the dbt workflow, enabling teams to collaborate effectively and maintain a history of changes to their dbt projects. + +In dbt Cloud, you can configure Git integrations to manage your dbt project code with ease. dbt Cloud offers multiple ways to integrate with you Git provider, catering to diverse team needs and preferences. + +Whether you use a Git integration that natively connects with dbt Cloud or prefer to work with a managed or cloned repository, dbt Cloud supports flexible options to streamline your workflow. + +
+ + -
-
-
-
\ No newline at end of file +
diff --git a/website/docs/docs/cloud/git/import-a-project-by-git-url.md b/website/docs/docs/cloud/git/import-a-project-by-git-url.md index 2b499b39cb7..804b1542e80 100644 --- a/website/docs/docs/cloud/git/import-a-project-by-git-url.md +++ b/website/docs/docs/cloud/git/import-a-project-by-git-url.md @@ -1,8 +1,6 @@ --- -title: "Import a project by git URL" -id: "import-a-project-by-git-url" -pagination_next: "docs/cloud/git/connect-github" -pagination_prev: null +title: "Connect with Git clone" +description: "Learn how to connect to a git repository using a git URL." --- In dbt Cloud, you can import a git repository from any valid git URL that points to a dbt project. There are some important considerations to keep in mind when doing this. @@ -10,7 +8,7 @@ In dbt Cloud, you can import a git repository from any valid git URL that points ## Git protocols You must use the `git@...` or `ssh:..`. version of your git URL, not the `https://...` version. dbt Cloud uses the SSH protocol to clone repositories, so dbt Cloud will be unable to clone repos supplied with the HTTP protocol. -## Managing Deploy Keys +## Managing deploy keys After importing a project by Git URL, dbt Cloud will generate a Deploy Key for your repository. To find the deploy key in dbt Cloud: diff --git a/website/docs/docs/collaborate/git/managed-repository.md b/website/docs/docs/cloud/git/managed-repository.md similarity index 83% rename from website/docs/docs/collaborate/git/managed-repository.md rename to website/docs/docs/cloud/git/managed-repository.md index db8e9840ccd..ba126c93dec 100644 --- a/website/docs/docs/collaborate/git/managed-repository.md +++ b/website/docs/docs/cloud/git/managed-repository.md @@ -1,6 +1,9 @@ --- -title: "Managed repository" +title: "Connect with managed repository" id: "managed-repository" +description: "Learn how to set up a project with a managed repository." +pagination_next: "docs/cloud/git/import-a-project-by-git-url" +pagination_prev: "docs/cloud/git/git-configuration-in-dbt-cloud" --- If you do not already have a git repository for your dbt project, you can let dbt Cloud manage a repository for you. Managed repositories are a great way to trial dbt without needing to create a new repository. diff --git a/website/docs/docs/collaborate/git/pr-template.md b/website/docs/docs/collaborate/git/pr-template.md index b621e31344d..c14b5e9dd24 100644 --- a/website/docs/docs/collaborate/git/pr-template.md +++ b/website/docs/docs/collaborate/git/pr-template.md @@ -14,7 +14,7 @@ The PR Template URL setting will be automatically set for most repositories, dep - If you connect to your repository via in-app integrations with your git provider or the "Git Clone" method via SSH, this URL setting will be auto-populated and editable. - For AWS CodeCommit, this URL setting isn't auto-populated and must be [manually configured](/docs/cloud/git/import-a-project-by-git-url#step-5-configure-pull-request-template-urls-optional). -- If you connect via a dbt Cloud [Managed repository](/docs/collaborate/git/managed-repository), this URL will not be set, and the IDE will prompt users to merge the changes directly into their default branch. +- If you connect via a dbt Cloud [Managed repository](/docs/cloud/git/managed-repository), this URL will not be set, and the IDE will prompt users to merge the changes directly into their default branch. The PR template URL supports two variables that can be used to build a URL string. These variables, `{{source}}` and `{{destination}}` return branch names based on the diff --git a/website/docs/faqs/Git/gitignore.md b/website/docs/faqs/Git/gitignore.md index f5892b30b83..350fd1e77c9 100644 --- a/website/docs/faqs/Git/gitignore.md +++ b/website/docs/faqs/Git/gitignore.md @@ -7,9 +7,11 @@ id: gitignore A `.gitignore` file specifies which files git should intentionally ignore or 'untrack'. dbt Cloud indicates untracked files in the project file explorer pane by putting the file or folder name in *italics*. -If you encounter issues like problems reverting changes, checking out or creating a new branch, or not being prompted to open a pull request after a commit in the dbt Cloud IDE — this usually indicates a problem with the [.gitignore](https://github.com/dbt-labs/dbt-starter-project/blob/main/.gitignore) file. The file may be missing or lacks the required entries for dbt Cloud to work correctly. +If you encounter issues like problems reverting changes, checking out or creating a new branch, or not being prompted to open a pull request after a commit in the dbt Cloud IDE — this usually indicates a problem with the [.gitignore](https://github.com/dbt-labs/dbt-starter-project/blob/main/.gitignore) file. The file may be missing or lacks the required entries for dbt Cloud to work correctly. -### Fix in the dbt Cloud IDE +The following sections describe how to fix the `.gitignore` file in: + + To resolve issues with your `gitignore` file, adding the correct entries won't automatically remove (or 'untrack') files or folders that have already been tracked by git. The updated `gitignore` will only prevent new files or folders from being tracked. So you'll need to first fix the `gitignore` file, then perform some additional git operations to untrack any incorrect files or folders. @@ -51,7 +53,9 @@ For more info on `gitignore` syntax, refer to the [Git docs](https://git-scm.com -### Fix in the git provider + + + Sometimes it's necessary to use the git providers web interface to fix a broken `.gitignore` file. Although the specific steps may vary across providers, the general process remains the same. @@ -121,3 +125,4 @@ dbt_modules/ For more info, refer to this [detailed video](https://www.loom.com/share/9b3b8e2b617f41a8bad76ec7e42dd014) for additional guidance. + diff --git a/website/docs/faqs/Troubleshooting/ide-session-unknown-error.md b/website/docs/faqs/Troubleshooting/ide-session-unknown-error.md index 4165506993c..5909459fbea 100644 --- a/website/docs/faqs/Troubleshooting/ide-session-unknown-error.md +++ b/website/docs/faqs/Troubleshooting/ide-session-unknown-error.md @@ -13,7 +13,7 @@ Your IDE session experienced an unknown error and was terminated. Please contact ``` -You can try to resolve this by adding a repository like a [managed repository](/docs/collaborate/git/managed-repository) or your preferred Git account. To add your Git account, navigate to **Project** > **Repository** and select your repository. +You can try to resolve this by adding a repository like a [managed repository](/docs/cloud/git/managed-repository) or your preferred Git account. To add your Git account, navigate to **Project** > **Repository** and select your repository. If you're still running into this error, please contact the Support team at support@getdbt.com for help. diff --git a/website/docs/guides/dbt-python-snowpark.md b/website/docs/guides/dbt-python-snowpark.md index 091f1006992..5f14222d910 100644 --- a/website/docs/guides/dbt-python-snowpark.md +++ b/website/docs/guides/dbt-python-snowpark.md @@ -262,7 +262,7 @@ We need to obtain our data source by copying our Formula 1 data into Snowflake t ## Configure dbt Cloud -1. We are going to be using [Snowflake Partner Connect](https://docs.snowflake.com/en/user-guide/ecosystem-partner-connect.html) to set up a dbt Cloud account. Using this method will allow you to spin up a fully fledged dbt account with your [Snowflake connection](/docs/cloud/connect-data-platform/connect-snowflake), [managed repository](/docs/collaborate/git/managed-repository), environments, and credentials already established. +1. We are going to be using [Snowflake Partner Connect](https://docs.snowflake.com/en/user-guide/ecosystem-partner-connect.html) to set up a dbt Cloud account. Using this method will allow you to spin up a fully fledged dbt account with your [Snowflake connection](/docs/cloud/connect-data-platform/connect-snowflake), [managed repository](/docs/cloud/git/managed-repository), environments, and credentials already established. 2. Navigate out of your worksheet back by selecting **home**. 3. In Snowsight, confirm that you are using the **ACCOUNTADMIN** role. 4. Navigate to the **Data Products** **> Partner Connect**. Find **dbt** either by using the search bar or navigating the **Data Integration**. Select the **dbt** tile. @@ -282,7 +282,7 @@ We need to obtain our data source by copying our Formula 1 data into Snowflake t 9. Select **Complete Registration**. You should now be redirected to your dbt Cloud account, complete with a connection to your Snowflake account, a deployment and a development environment, and a sample job. -10. To help you version control your dbt project, we have connected it to a [managed repository](/docs/collaborate/git/managed-repository), which means that dbt Labs will be hosting your repository for you. This will give you access to a Git workflow without you having to create and host the repository yourself. You will not need to know Git for this workshop; dbt Cloud will help guide you through the workflow. In the future, when you’re developing your own project, [feel free to use your own repository](/docs/cloud/git/connect-github). This will allow you to learn more about features like [Slim CI](/docs/deploy/continuous-integration) builds after this workshop. +10. To help you version control your dbt project, we have connected it to a [managed repository](/docs/cloud/git/managed-repository), which means that dbt Labs will be hosting your repository for you. This will give you access to a Git workflow without you having to create and host the repository yourself. You will not need to know Git for this workshop; dbt Cloud will help guide you through the workflow. In the future, when you’re developing your own project, [feel free to use your own repository](/docs/cloud/git/connect-github). This will allow you to learn more about features like [Slim CI](/docs/deploy/continuous-integration) builds after this workshop. ## Change development schema name navigate the IDE diff --git a/website/docs/guides/sl-snowflake-qs.md b/website/docs/guides/sl-snowflake-qs.md index 79038cd1dfc..d70d074485f 100644 --- a/website/docs/guides/sl-snowflake-qs.md +++ b/website/docs/guides/sl-snowflake-qs.md @@ -262,7 +262,7 @@ There are two ways to connect dbt Cloud to Snowflake. The first option is Partne -Using Partner Connect allows you to create a complete dbt account with your [Snowflake connection](/docs/cloud/connect-data-platform/connect-snowflake), [a managed repository](/docs/collaborate/git/managed-repository), [environments](/docs/build/custom-schemas#managing-environments), and credentials. +Using Partner Connect allows you to create a complete dbt account with your [Snowflake connection](/docs/cloud/connect-data-platform/connect-snowflake), [a managed repository](/docs/cloud/git/managed-repository), [environments](/docs/build/custom-schemas#managing-environments), and credentials. 1. In the Snowflake UI, click on the home icon in the upper left corner. In the left sidebar, select **Data Products**. Then, select **Partner Connect**. Find the dbt tile by scrolling or by searching for dbt in the search bar. Click the tile to connect to dbt. @@ -333,7 +333,7 @@ Using Partner Connect allows you to create a complete dbt account with your [Sno ## Set up a dbt Cloud managed repository -If you used Partner Connect, you can skip to [initializing your dbt project](#initialize-your-dbt-project-and-start-developing) as Partner Connect provides you with a [managed repository](/docs/collaborate/git/managed-repository). Otherwise, you will need to create your repository connection. +If you used Partner Connect, you can skip to [initializing your dbt project](#initialize-your-dbt-project-and-start-developing) as Partner Connect provides you with a [managed repository](/docs/cloud/git/managed-repository). Otherwise, you will need to create your repository connection. diff --git a/website/docs/guides/snowflake-qs.md b/website/docs/guides/snowflake-qs.md index f1edd5ffc00..40bdeed1ef2 100644 --- a/website/docs/guides/snowflake-qs.md +++ b/website/docs/guides/snowflake-qs.md @@ -142,7 +142,7 @@ There are two ways to connect dbt Cloud to Snowflake. The first option is Partne -Using Partner Connect allows you to create a complete dbt account with your [Snowflake connection](/docs/cloud/connect-data-platform/connect-snowflake), [a managed repository](/docs/collaborate/git/managed-repository), [environments](/docs/build/custom-schemas#managing-environments), and credentials. +Using Partner Connect allows you to create a complete dbt account with your [Snowflake connection](/docs/cloud/connect-data-platform/connect-snowflake), [a managed repository](/docs/cloud/git/managed-repository), [environments](/docs/build/custom-schemas#managing-environments), and credentials. 1. In the Snowflake UI, click on the home icon in the upper left corner. In the left sidebar, select **Data Products**. Then, select **Partner Connect**. Find the dbt tile by scrolling or by searching for dbt in the search bar. Click the tile to connect to dbt. diff --git a/website/sidebars.js b/website/sidebars.js index db97f1f25da..3a8f560c297 100644 --- a/website/sidebars.js +++ b/website/sidebars.js @@ -142,6 +142,7 @@ const sidebarSettings = { }, items: [ "docs/cloud/git/git-configuration-in-dbt-cloud", + "docs/cloud/git/managed-repository", "docs/cloud/git/import-a-project-by-git-url", "docs/cloud/git/connect-github", "docs/cloud/git/connect-gitlab", @@ -552,7 +553,6 @@ const sidebarSettings = { items: [ "docs/collaborate/git-version-control", "docs/collaborate/git/version-control-basics", - "docs/collaborate/git/managed-repository", "docs/collaborate/git/pr-template", "docs/collaborate/git/merge-conflicts", ], diff --git a/website/snippets/available-git-providers.md b/website/snippets/available-git-providers.md index 6579d8989bf..5e1200fe2f3 100644 --- a/website/snippets/available-git-providers.md +++ b/website/snippets/available-git-providers.md @@ -1,3 +1,3 @@ When you develop in dbt Cloud, you can leverage [Git](/docs/collaborate/git-version-control) to version control your code. -To connect to a repository, you can either set up a dbt Cloud-hosted [managed repository](/docs/collaborate/git/managed-repository) or directly connect to a [supported git provider](/docs/cloud/git/connect-github). Managed repositories are a great way to trial dbt without needing to create a new repository. In the long run, it's better to connect to a supported git provider to use features like automation and [continuous integration](/docs/deploy/continuous-integration). \ No newline at end of file +To connect to a repository, you can either set up a dbt Cloud-hosted [managed repository](/docs/cloud/git/managed-repository) or directly connect to a [supported git provider](/docs/cloud/git/connect-github). Managed repositories are a great way to trial dbt without needing to create a new repository. In the long run, it's better to connect to a supported git provider to use features like automation and [continuous integration](/docs/deploy/continuous-integration). diff --git a/website/vercel.json b/website/vercel.json index b68dc053db9..993ff9065bd 100644 --- a/website/vercel.json +++ b/website/vercel.json @@ -2,6 +2,11 @@ "cleanUrls": true, "trailingSlash": false, "redirects": [ + { + "source": "/docs/collaborate/git/managed-repository", + "destination": "/docs/cloud/git/managed-repository", + "permanent": true + }, { "source": "/faqs/API/rotate-token", "destination": "/docs/dbt-cloud-apis/service-tokens#service-token-update", @@ -1372,7 +1377,7 @@ }, { "source": "/docs/dbt-cloud/cloud-configuring-dbt-cloud/cloud-using-a-managed-repository", - "destination": "/docs/collaborate/git/managed-repository", + "destination": "/docs/cloud/git/managed-repository", "permanent": true }, { From 512109ad0422878a850cbd1b656792ae76c03ee6 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Thu, 19 Dec 2024 13:35:11 +0000 Subject: [PATCH 22/81] Update managed-repository.md --- website/docs/docs/cloud/git/managed-repository.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/website/docs/docs/cloud/git/managed-repository.md b/website/docs/docs/cloud/git/managed-repository.md index ba126c93dec..ef10d6324a7 100644 --- a/website/docs/docs/cloud/git/managed-repository.md +++ b/website/docs/docs/cloud/git/managed-repository.md @@ -8,6 +8,10 @@ pagination_prev: "docs/cloud/git/git-configuration-in-dbt-cloud" If you do not already have a git repository for your dbt project, you can let dbt Cloud manage a repository for you. Managed repositories are a great way to trial dbt without needing to create a new repository. +:::info +We do not recommend using a managed repository in a production environment. You will not be able to use git features like pull requests which are part of our recommended version control best practices. +::: + To set up a project with a managed repository: 1. From your Account settings in dbt Cloud, select the project you want to set up with a managed repository. If the project already has a repository set up, you need to edit the repository settings and disconnect the existing repository. @@ -18,6 +22,4 @@ To set up a project with a managed repository: 6. Click **Create**. -dbt Cloud will host and manage this repository for you. If in the future you choose to host this repository elsewhere, you can export the information from dbt Cloud at any time. - -** We do not recommend using a managed repository in a production environment. You will not be able to use git features like pull requests which are part of our recommended version control best practices. +dbt Cloud will host and manage this repository for you. If in the future you choose to host this repository elsewhere, you can export the information from dbt Cloud at any time. Refer to [Move from a managed repository to a self-hosted repository](/faqs/Git/managed-repo) for more information. From 8b331eea0b5f4df16f54b4d70454f12d7798de04 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Fri, 20 Dec 2024 15:59:36 +0000 Subject: [PATCH 23/81] add callouts about email mismatch --- .../cloud/manage-access/external-oauth.md | 50 ++++++++++++------- 1 file changed, 31 insertions(+), 19 deletions(-) diff --git a/website/docs/docs/cloud/manage-access/external-oauth.md b/website/docs/docs/cloud/manage-access/external-oauth.md index 380d0a3d1cc..8f434657ad2 100644 --- a/website/docs/docs/cloud/manage-access/external-oauth.md +++ b/website/docs/docs/cloud/manage-access/external-oauth.md @@ -29,6 +29,11 @@ The process of setting up external OAuth will require a little bit of back-and-f If the admins that handle these products are all different people, it’s better to have them coordinating simultaneously to reduce friction. +:::info Username and credential matching required +Ensure that the username entered by the IdP admin matches the username in the Snowflake credentials. If the email address used in the dbt Cloud setup is different from the Snowflake username, the connection will fail or you may run into issues. +::: + + ### Snowflake commands The following is a template for creating the OAuth configurations in the Snowflake environment: @@ -45,18 +50,22 @@ external_oauth_audience_list = ('') external_oauth_token_user_mapping_claim = 'sub' external_oauth_snowflake_user_mapping_attribute = 'email_address' external_oauth_any_role_mode = 'ENABLE' - ``` The `external_oauth_token_user_mapping_claim` and `external_oauth_snowflake_user_mapping_attribute` can be modified based on the your organizations needs. These values point to the claim in the users’ token. In the example, Snowflake will look up the Snowflake user whose `email` matches the value in the `sub` claim. -**Note:** The Snowflake default roles ACCOUNTADMIN, ORGADMIN, or SECURITYADMIN, are blocked from external OAuth by default and they will likely fail to authenticate. See the [Snowflake documentation](https://docs.snowflake.com/en/sql-reference/sql/create-security-integration-oauth-external) for more information. +**Notes:** +- The Snowflake default roles ACCOUNTADMIN, ORGADMIN, or SECURITYADMIN, are blocked from external OAuth by default and they will likely fail to authenticate. See the [Snowflake documentation](https://docs.snowflake.com/en/sql-reference/sql/create-security-integration-oauth-external) for more information. +- The value for `external_oauth_snowflake_user_mapping_attribute` must map correctly to the Snowflake username. For example, if `email_address` is used, the email in the token from the IdP must match the Snowflake username exactly. ## Identity provider configuration -Select a supported identity provider (IdP) for instructions on configuring external OAuth in their environment and completing the integration in dbt Cloud. +Select a supported identity provider (IdP) for instructions on configuring external OAuth in their environment and completing the integration in dbt Cloud: - +- [Okta](#okta) +- [Entra ID](#entra-id) + +## Okta ### 1. Initialize the dbt Cloud settings @@ -139,6 +148,11 @@ Adjust the other settings as needed to meet your organization's configurations i 3. Run the steps to create the integration in Snowflake. +:::info Username consistency +Ensure that the username (for example, email address) entered in the IdP matches the Snowflake credentials for all users. Mismatched usernames will result in authentication failures. +::: + + ### 5. Configuring the integration in dbt Cloud 1. Navigate back to the dbt Cloud **Account settings** —> **Integrations** page you were on at the beginning. It’s time to start filling out all of the fields. @@ -169,9 +183,9 @@ Adjust the other settings as needed to meet your organization's configurations i 4. **Save** the connection, and you have now configured External OAuth with Okta and Snowflake! - - + +## Entra ID ### 1. Initialize the dbt Cloud settings @@ -179,21 +193,19 @@ Adjust the other settings as needed to meet your organization's configurations i 2. Scroll down to **Custom integrations** and click **Add integrations**. 3. Leave this window open. You can set the **Integration type** to Entra ID and note the **Redirect URI** at the bottom of the page. Copy this to your clipboard for use in the next steps. -### Entra ID +### 2. Create the Entra ID apps -You’ll create two apps in the Azure portal: A resource server and a client app. +- You’ll create two apps in the Azure portal: A resource server and a client app. +- In your Azure portal, open the **Entra ID** and click **App registrations** from the left menu. :::important -The admin who creates the apps in the Microsoft Entra ID account must also be a user in Snowflake. - -The `value` field gathered in these steps is only displayed once. When created, record it immediately. - +- The admin who creates the apps in the Microsoft Entra ID account must also be a user in Snowflake. +- The `value` field gathered in these steps is only displayed once. When created, record it immediately. +- Ensure that the username (for example, email address) entered in the IdP matches the Snowflake credentials for all users. Mismatched usernames will result in authentication failures. ::: -In your Azure portal, open the **Entra ID** and click **App registrations** from the left menu. - -### 1. Create a resource server +### 3. Create a resource server 1. From the app registrations screen, click **New registration**. 1. Give the app a name. @@ -209,7 +221,7 @@ In your Azure portal, open the **Entra ID** and click **App registrations** from 4. Ensure **State** is set to **Enabled**. 5. Click **Add scope**. -### 2. Create a client app +### 4. Create a client app 1. From the **App registration page**, click **New registration**. 1. Give the app a name that uniquely identifies it as the client app. @@ -225,7 +237,7 @@ In your Azure portal, open the **Entra ID** and click **App registrations** from 7. Record the `value` for use in a future step and record it immediately. **Note**: Entra ID will not display this value again once you navigate away from this screen. -### 3. Snowflake configuration +### 5. Snowflake configuration You'll be switching between the Entra ID site and Snowflake. Keep your Entra ID account open for this process. @@ -257,7 +269,7 @@ app in Entra ID, click **Endpoints** and open the **Federation metadata document - The **Application ID URI** maps to the `external_oauth_audience_list` field in Snowflake. 4. Run the configurations. Be sure the admin who created the Microsoft apps is also a user in Snowflake, or the configuration will fail. -### 4. Configuring the integration in dbt Cloud +### 6. Configuring the integration in dbt Cloud 1. Navigate back to the dbt Cloud **Account settings** —> **Integrations** page you were on at the beginning. It’s time to start filling out all of the fields. There will be some back-and-forth between the Entra ID account and dbt Cloud. 2. `Integration name`: Give the integration a descriptive name that includes identifying information about the Entra ID environment so future users won’t have to guess where it belongs. @@ -266,7 +278,7 @@ app in Entra ID, click **Endpoints** and open the **Federation metadata document 5. `Authorization URL` and `Token URL`: From the client ID app, open the `Endpoints` tab. These URLs map to the `OAuth 2.0 authorization endpoint (v2)` and `OAuth 2.0 token endpoint (v2)` fields. *You must use v2 of the `OAuth 2.0 authorization endpoint`. Do not use V1.* You can use either version of the `OAuth 2.0 token endpoint`. 6. `Application ID URI`: Copy the `Application ID URI` field from the resource server’s Overview screen. - + ## FAQs From 38013cc3dc25e592651d84634e54b5a3b2bbef3c Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 20 Dec 2024 16:59:58 +0000 Subject: [PATCH 24/81] remove description value from model notifications example this pr removes the `description` field/value from the groups.yml example. this is to make sure it isn't confused with the `description` config that dbt uses. pr also adds a callout clarifying that the `owner` field accepts any input (liek favorite_food). --- website/docs/docs/deploy/model-notifications.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/website/docs/docs/deploy/model-notifications.md b/website/docs/docs/deploy/model-notifications.md index 45ffbef0a4f..7396f67f14d 100644 --- a/website/docs/docs/deploy/model-notifications.md +++ b/website/docs/docs/deploy/model-notifications.md @@ -38,21 +38,29 @@ groups: - name: finance owner: # Email is required to receive model-level notifications, additional properties are also allowed. - name: "Finance Team" - description: "Models related to the finance department" + name: "Finance team" email: finance@dbtlabs.com favorite_food: donuts - name: marketing owner: - name: "Marketing Team" - description: "Models related to the marketing department" + name: "Marketing team" email: marketing@dbtlabs.com favorite_food: jaffles + + - name: docs + owner: + name: "Documentation team" + email: docs@dbtlabs.com + favorite_food: pizza ``` +:::tip +The `owner` value is flexible and accepts arbitrary inputs in addition to the required `email` field. For example, you could include a custom field like `favorite_food` to add context about the team. +::: + ## Attach groups to models Attach groups to models as you would any other config, in either the `dbt_project.yml` or `whatever.yml` files. For example: From cb4e20bb20f09c73f212fe54365bb461d4d60cdf Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Fri, 20 Dec 2024 17:00:44 +0000 Subject: [PATCH 25/81] Update website/docs/docs/deploy/model-notifications.md --- website/docs/docs/deploy/model-notifications.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/deploy/model-notifications.md b/website/docs/docs/deploy/model-notifications.md index 7396f67f14d..2a2dbc45e78 100644 --- a/website/docs/docs/deploy/model-notifications.md +++ b/website/docs/docs/deploy/model-notifications.md @@ -58,7 +58,7 @@ groups: :::tip -The `owner` value is flexible and accepts arbitrary inputs in addition to the required `email` field. For example, you could include a custom field like `favorite_food` to add context about the team. +The `owner` key is flexible and accepts arbitrary inputs in addition to the required `email` field. For example, you could include a custom field like `favorite_food` to add context about the team. ::: ## Attach groups to models From 3cd8a0e7fcdd1dac24b193198917a54222bbb24c Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 23 Dec 2024 10:10:01 +0000 Subject: [PATCH 26/81] add explorer udpates this pr updates the explorer doc so it includes the recent changes: - support for saved queries - support for 'contains' logic in search --- website/docs/docs/collaborate/explore-projects.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/website/docs/docs/collaborate/explore-projects.md b/website/docs/docs/collaborate/explore-projects.md index 0e4dd7963c8..4068149187d 100644 --- a/website/docs/docs/collaborate/explore-projects.md +++ b/website/docs/docs/collaborate/explore-projects.md @@ -113,7 +113,7 @@ Lenses are helpful to analyze a subset of the DAG if you're zoomed in, or to fin A resource in your project is characterized by resource type, materialization type, or model layer, as well as its latest run or latest test status. Lenses are available for the following metadata: -- **Relationship**: Organizes resources by resource type, such as models, tests, seeds, and [more](/reference/node-selection/syntax). Resource type uses the `resource_type` selector. +- **Resource type**: Organizes resources by resource type, such as models, tests, seeds, saved query, and [more](/docs/build/projects). Resource type uses the `resource_type` selector. - **Materialization type**: Identifies the strategy for building the dbt models in your data platform. - **Latest status**: The status from the latest execution of the resource in the current environment. For example, diagnosing a failed DAG region. - **Model layer**: The modeling layer that the model belongs to according to [best practices guide](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview#guide-structure-overview). For example, discovering marts models to analyze. @@ -121,13 +121,13 @@ A resource in your project is characterized by resource type, materialization ty - **Intermediate** — A model with the prefix `int_`. Or, a model that lives in the `/int/` or `/intermediate/` subdirectory. - **Staging** — A model with the prefix `stg_`. Or, a model that lives in the `/staging/` subdirectory. - **Test status**: The status from the latest execution of the tests that ran again this resource. In the case that a model has multiple tests with different results, the lens reflects the 'worst case' status. -- **Usage queries**: The number of queries against this resource over a given time period. +- **Consumption query history**: The number of queries against this resource over a given time period. ### Example of lenses -Example of applying the **Materialization Type** _lens_ with the lineage graph zoomed out. In this view, each model name has a color according to the materialization type legend at the bottom, which specifies the materialization type. This color-coding helps to quickly identify the materialization types of different models. +Example of applying the **Materialization type** _lens_ with the lineage graph zoomed out. In this view, each model name has a color according to the materialization type legend at the bottom, which specifies the materialization type. This color-coding helps to quickly identify the materialization types of different models. @@ -141,7 +141,7 @@ You can locate resources in your project by performing a keyword search in the s -- **Partial keyword search** — This is also referred to as fuzzy search. +- **Partial keyword search** — Also referred to as fuzzy search. Explorer also uses a "contains" logic to improve your search results. This means you don't need to know the exact root word of your search term. - **Exclude keywords** — Prepend a minus sign (-) to the keyword you want to exclude from search results. For example, `-user` will exclude all matches of that keyword from search results. - **Boolean operators** — Use Boolean operators to enhance your keyword search. For example, the search results for `users OR github` will include matches for either keyword. - **Phrase search** — Surround a string of keywords with double quotation marks to search for that exact phrase (for example, `"stg users"`). To learn more, refer to [Phrase search](https://en.wikipedia.org/wiki/Phrase_search) on Wikipedia. From 952374035b02b072216dfefcc7bb499b5edbb74e Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Mon, 23 Dec 2024 12:07:35 +0000 Subject: [PATCH 27/81] feedback --- website/docs/reference/resource-configs/tags.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/website/docs/reference/resource-configs/tags.md b/website/docs/reference/resource-configs/tags.md index 45c4e719059..7e0016f0d35 100644 --- a/website/docs/reference/resource-configs/tags.md +++ b/website/docs/reference/resource-configs/tags.md @@ -22,7 +22,7 @@ datatype: string | [string] [models](/reference/model-configs): [](/reference/resource-configs/resource-path): - +tags: | [] + +tags: | [] # Supports single strings or list of strings [snapshots](/reference/snapshot-configs): [](/reference/resource-configs/resource-path): @@ -41,7 +41,7 @@ datatype: string | [string] [models](/reference/model-configs): [](/reference/resource-configs/resource-path): - +tags: | [] + +tags: | [] # Supports single strings or list of strings [snapshots](/reference/snapshot-configs): [](/reference/resource-configs/resource-path): @@ -80,15 +80,16 @@ The following examples show how to add tags to dbt resources in YAML files. Repl resource_type: - name: resource_name config: - tags: string - # Optional: Add specific properties for models + tags: | [] # Supports single strings or list of strings + # Optional: Add the following specific properties for models columns: - name: column_name - tags: string + tags: | [] tests: test-name: config: - tags: string + tags: "single-string" # Supports single string + tags: ["string-1", "string-2"] # Supports list of strings ``` From 90d624caa2632f9ba7fa40eb7cd8e32589b45217 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 23 Dec 2024 16:20:20 +0000 Subject: [PATCH 28/81] Update website/docs/docs/collaborate/explore-projects.md --- website/docs/docs/collaborate/explore-projects.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/collaborate/explore-projects.md b/website/docs/docs/collaborate/explore-projects.md index 4068149187d..fb16c89ef39 100644 --- a/website/docs/docs/collaborate/explore-projects.md +++ b/website/docs/docs/collaborate/explore-projects.md @@ -141,7 +141,7 @@ You can locate resources in your project by performing a keyword search in the s -- **Partial keyword search** — Also referred to as fuzzy search. Explorer also uses a "contains" logic to improve your search results. This means you don't need to know the exact root word of your search term. +- **Partial keyword search** — Also referred to as fuzzy search. Explorer uses a "contains" logic to improve your search results. This means you don't need to know the exact root word of your search term. - **Exclude keywords** — Prepend a minus sign (-) to the keyword you want to exclude from search results. For example, `-user` will exclude all matches of that keyword from search results. - **Boolean operators** — Use Boolean operators to enhance your keyword search. For example, the search results for `users OR github` will include matches for either keyword. - **Phrase search** — Surround a string of keywords with double quotation marks to search for that exact phrase (for example, `"stg users"`). To learn more, refer to [Phrase search](https://en.wikipedia.org/wiki/Phrase_search) on Wikipedia. From b214a2411727e921fe512c57f93461d1d6b62a37 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 23 Dec 2024 22:05:57 +0000 Subject: [PATCH 29/81] clarify full-refresh behavior (#6700) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit this pr clarifies full_refresh config behavior when you set it to `false` and `true`. Resolves #6698 --- 🚀 Deployment available! Here are the direct links to the updated files: - https://docs-getdbt-com-git-update-full-refresh-dbt-labs.vercel.app/docs/build/incremental-microbatch - https://docs-getdbt-com-git-update-full-refresh-dbt-labs.vercel.app/reference/resource-configs/full_refresh --------- Co-authored-by: Leona B. Campbell <3880403+runleonarun@users.noreply.github.com> --- .../docs/docs/build/incremental-microbatch.md | 2 +- .../resource-configs/full_refresh.md | 35 +++++++++++-------- 2 files changed, 21 insertions(+), 16 deletions(-) diff --git a/website/docs/docs/build/incremental-microbatch.md b/website/docs/docs/build/incremental-microbatch.md index 4aff8b5839c..77f9ac274b0 100644 --- a/website/docs/docs/build/incremental-microbatch.md +++ b/website/docs/docs/build/incremental-microbatch.md @@ -232,7 +232,7 @@ from {{ source('sales', 'transactions') }} ### Full refresh -As a best practice, we recommend configuring `full_refresh: False` on microbatch models so that they ignore invocations with the `--full-refresh` flag. If you need to reprocess historical data, do so with a targeted backfill that specifies explicit start and end dates. +As a best practice, we recommend [configuring `full_refresh: false`](/reference/resource-configs/full_refresh) on microbatch models so that they ignore invocations with the `--full-refresh` flag. If you need to reprocess historical data, do so with a targeted backfill that specifies explicit start and end dates. ## Usage diff --git a/website/docs/reference/resource-configs/full_refresh.md b/website/docs/reference/resource-configs/full_refresh.md index c874fe7a396..5e291fa2454 100644 --- a/website/docs/reference/resource-configs/full_refresh.md +++ b/website/docs/reference/resource-configs/full_refresh.md @@ -4,6 +4,8 @@ description: "Setting the full_refresh config to false prevents a model or seed datatype: boolean --- +The `full_refresh` config allows you to control whether a resource will always or never perform a full-refresh. This config overrides the `--full-refresh` command-line flag. + ](/reference/resource-configs/resource-path): - +full_refresh: false - + +full_refresh: false | true ``` @@ -30,17 +31,14 @@ models: ```sql {{ config( - full_refresh = false + full_refresh = false | true ) }} select ... - ``` -The configured model(s) will not full-refresh when `dbt run --full-refresh` is invoked. - @@ -50,27 +48,34 @@ The configured model(s) will not full-refresh when `dbt run --full-refresh` is i ```yml seeds: [](/reference/resource-configs/resource-path): - +full_refresh: false + +full_refresh: false | true ``` -The configured seed(s) will not full-refresh when `dbt seed --full-refresh` is invoked. - +- If `full_refresh:true` — the configured resources(s) will full-refresh when `dbt run --full-refresh` is invoked. +- If `full_refresh:false` — the configured resources(s) will _not_ full-refresh when `dbt run --full-refresh` is invoked. + + ## Description -Optionally set a resource to always or never full-refresh. -- If specified as `true` or `false`, the -`full_refresh` config will take precedence over the presence or absence of the `--full-refresh` flag. -- If the `full_refresh` config is `none` or omitted, the resource will use the value of the `--full-refresh` flag. -**Note:** The `--full-refresh` flag also supports a short name, `-f`. +The `full_refresh` config allows you to optionally configure whether a resource will always or never perform a full-refresh. This config is an override for the `--full-refresh` command line flag used when running dbt commands. + + +| `full_refresh` value | Behavior | +| ---------------------------- | -------- | +| `true` | The resource always full-refreshes, regardless of the presence or absence of the `--full-refresh` flag. | +| `false` | The resource never full-refreshes, even if the `--full-refresh` flag is provided. | +| `none` or omitted | The resource follows the behavior of the `--full-refresh` flag. If the flag is used, the resource will full-refresh; otherwise, it won't. | -This logic is encoded in the [`should_full_refresh()`](https://github.com/dbt-labs/dbt-adapters/blob/60005a0a2bd33b61cb65a591bc1604b1b3fd25d5/dbt/include/global_project/macros/materializations/configs.sql) macro. +#### Note +- The `--full-refresh` flag also supports a short name, `-f`. +- The [`should_full_refresh()`](https://github.com/dbt-labs/dbt-adapters/blob/60005a0a2bd33b61cb65a591bc1604b1b3fd25d5/dbt/include/global_project/macros/materializations/configs.sql) macro has logic encoded. ## Usage From 7224646ffea749b7472a7d21bfcf3018f12ddbe5 Mon Sep 17 00:00:00 2001 From: "Leona B. Campbell" <3880403+runleonarun@users.noreply.github.com> Date: Mon, 23 Dec 2024 16:31:18 -0800 Subject: [PATCH 30/81] Update website/docs/docs/collaborate/explore-projects.md --- website/docs/docs/collaborate/explore-projects.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/collaborate/explore-projects.md b/website/docs/docs/collaborate/explore-projects.md index fb16c89ef39..627b255cd78 100644 --- a/website/docs/docs/collaborate/explore-projects.md +++ b/website/docs/docs/collaborate/explore-projects.md @@ -141,7 +141,7 @@ You can locate resources in your project by performing a keyword search in the s -- **Partial keyword search** — Also referred to as fuzzy search. Explorer uses a "contains" logic to improve your search results. This means you don't need to know the exact root word of your search term. +- **Partial keyword search** — Also referred to as fuzzy search. Explorer uses a "contains" logic to improve your search results. This means you can search for partial terms without knowing the exact root word of your search term. - **Exclude keywords** — Prepend a minus sign (-) to the keyword you want to exclude from search results. For example, `-user` will exclude all matches of that keyword from search results. - **Boolean operators** — Use Boolean operators to enhance your keyword search. For example, the search results for `users OR github` will include matches for either keyword. - **Phrase search** — Surround a string of keywords with double quotation marks to search for that exact phrase (for example, `"stg users"`). To learn more, refer to [Phrase search](https://en.wikipedia.org/wiki/Phrase_search) on Wikipedia. From cf53465ec65837849472a948dd2e92c20cd131dc Mon Sep 17 00:00:00 2001 From: Anks S <6273915+asarraf@users.noreply.github.com> Date: Mon, 23 Dec 2024 20:58:11 -0800 Subject: [PATCH 31/81] Update 08-upgrading-to-v1.7.md Correct the instructions for `dbt clean` section of Upgrading to 1.7 --- .../docs/docs/dbt-versions/core-upgrade/08-upgrading-to-v1.7.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/dbt-versions/core-upgrade/08-upgrading-to-v1.7.md b/website/docs/docs/dbt-versions/core-upgrade/08-upgrading-to-v1.7.md index df24b63a2f0..b98a76295cf 100644 --- a/website/docs/docs/dbt-versions/core-upgrade/08-upgrading-to-v1.7.md +++ b/website/docs/docs/dbt-versions/core-upgrade/08-upgrading-to-v1.7.md @@ -66,7 +66,7 @@ dbt Core v1.5 introduced model governance which we're continuing to refine. v1. ### dbt clean -Starting in v1.7, `dbt clean` will only clean paths within the current working directory. The `--no-clean-project-files-only` flag will delete all paths specified in `clean-paths`, even if they're outside the dbt project. +Starting in v1.7, `dbt clean` will only clean paths within the current working directory. The `--no-clean-project-files-only` flag will delete all paths specified in the `clean-targets` section of `dbt_project.yml`, even if they're outside the dbt project. Supported flags: - `--clean-project-files-only` (default) From 26e35aff3a4dda83a69071f90fb8899ceee1e84e Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Tue, 24 Dec 2024 09:37:22 +0000 Subject: [PATCH 32/81] update title --- website/docs/docs/cloud/manage-access/external-oauth.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/website/docs/docs/cloud/manage-access/external-oauth.md b/website/docs/docs/cloud/manage-access/external-oauth.md index 121ba8f4cb4..ccb8fd66cc3 100644 --- a/website/docs/docs/cloud/manage-access/external-oauth.md +++ b/website/docs/docs/cloud/manage-access/external-oauth.md @@ -22,18 +22,17 @@ dbt Cloud Enterprise supports [external OAuth authentication](https://docs.snow The process of setting up external OAuth will require a little bit of back-and-forth between your dbt Cloud, IdP, and Snowflake accounts, and having them open in multiple browser tabs will help speed up the configuration process: -- **dbt Cloud:** You’ll primarily be working in the **Account Settings** —> **Integrations** page. You will need [proper permission](/docs/cloud/manage-access/enterprise-permissions) to set up the integration and create the connections. +- **dbt Cloud:** You’ll primarily be working in the **Account settings** —> **Integrations** page. You will need [proper permission](/docs/cloud/manage-access/enterprise-permissions) to set up the integration and create the connections. - **Snowflake:** Open a worksheet in an account that has permissions to [create a security integration](https://docs.snowflake.com/en/sql-reference/sql/create-security-integration). - **Okta:** You’ll be working in multiple areas of the Okta account, but you can start in the **Applications** section. You will need permissions to [create an application](https://help.okta.com/en-us/content/topics/security/custom-admin-role/about-role-permissions.htm#Application_permissions) and an [authorization server](https://help.okta.com/en-us/content/topics/security/custom-admin-role/about-role-permissions.htm#Authorization_server_permissions). - **Entra ID** An admin with access to create [Entra ID apps](https://learn.microsoft.com/en-us/entra/identity/role-based-access-control/custom-available-permissions) who is also a user in Snowflake is required. If the admins that handle these products are all different people, it’s better to have them coordinating simultaneously to reduce friction. -:::info Username and credential matching required -Ensure that the username entered by the IdP admin matches the username in the Snowflake credentials. If the email address used in the dbt Cloud setup is different from the Snowflake username, the connection will fail or you may run into issues. +:::info Snowflake and IdP username matching required +Ensure that the username/email address entered by the IdP admin matches the Snowflake credentials username. If the email address used in the dbt Cloud setup is different from the Snowflake email address, the connection will fail or you may run into issues. ::: - ### Snowflake commands The following is a template for creating the OAuth configurations in the Snowflake environment: From 5fc9357903e14a525edc9c1a4f076f5090203f33 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Tue, 24 Dec 2024 10:42:29 +0000 Subject: [PATCH 33/81] clarify meta acceptance --- website/docs/reference/resource-configs/meta.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/resource-configs/meta.md b/website/docs/reference/resource-configs/meta.md index 7214dac560b..313595ea497 100644 --- a/website/docs/reference/resource-configs/meta.md +++ b/website/docs/reference/resource-configs/meta.md @@ -290,7 +290,7 @@ saved_queries: ## Definition -The `meta` field can be used to set metadata for a resource. This metadata is compiled into the `manifest.json` file generated by dbt, and is viewable in the auto-generated documentation. +The `meta` field can be used to set metadata for a resource and accepts any key-value pairs. This metadata is compiled into the `manifest.json` file generated by dbt, and is viewable in the auto-generated documentation. Depending on the resource you're configuring, `meta` may be available within the `config` property, and/or as a top-level key. (For backwards compatibility, `meta` is often (but not always) supported as a top-level key, though without the capabilities of config inheritance.) From 3e561f12b0b1e35a531e0c17363218d3beadf64b Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 24 Dec 2024 10:53:36 +0000 Subject: [PATCH 34/81] Update tags.md add command examples --- website/docs/reference/resource-configs/tags.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/website/docs/reference/resource-configs/tags.md b/website/docs/reference/resource-configs/tags.md index 7e0016f0d35..27be1f71766 100644 --- a/website/docs/reference/resource-configs/tags.md +++ b/website/docs/reference/resource-configs/tags.md @@ -114,6 +114,7 @@ Apply a tag (or list of tags) to a resource. These tags can be used as part of the [resource selection syntax](/reference/node-selection/syntax), when running the following commands: - `dbt run --select tag:my_tag` +- `dbt build --select tag:my_tag` - `dbt seed --select tag:my_tag` - `dbt snapshot --select tag:my_tag` - `dbt test --select tag:my_tag` (indirectly runs all tests associated with the models that are tagged) @@ -222,6 +223,13 @@ This following example shows how to apply a tag to a saved query in the `dbt_pro +Then, run part of your project like so: + +``` +# Run all resources tagged "order_metrics" +dbt run --select tag:order_metrics +``` + The second example shows how to apply multiple tags to a saved query in the `semantic_model.yml` file. The saved query is then tagged with `order_metrics` and `hourly`. @@ -236,8 +244,15 @@ saved_queries: - order_metrics - hourly ``` - + + +Then, run part of your project like so: + +``` +# Run all resources tagged "order_metrics" and "hourly" +dbt build --select tag:order_metrics tag:hourly +``` ## Usage notes From 41a5d19b6655834424d02e4b23921ac1c607a975 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 24 Dec 2024 10:57:30 +0000 Subject: [PATCH 35/81] Update meta.md add link --- website/docs/reference/resource-configs/meta.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/reference/resource-configs/meta.md b/website/docs/reference/resource-configs/meta.md index 313595ea497..a7f348d50ba 100644 --- a/website/docs/reference/resource-configs/meta.md +++ b/website/docs/reference/resource-configs/meta.md @@ -179,7 +179,7 @@ exposures: -Configure `meta` in the your semantic models YAML file or under the `semantic-models` config block in the `dbt_project.yml` file. +Configure `meta` in the your [semantic models](/docs/build/semantic-models) YAML file or under the `semantic-models` config block in the `dbt_project.yml` file. @@ -381,7 +381,7 @@ models: ### Assign meta to semantic model -The following example shows how to assign a `meta` value to a semantic model in the `semantic_model.yml` file and `dbt_project.yml` file: +The following example shows how to assign a `meta` value to a [semantic model](/docs/build/semantic-models) in the `semantic_model.yml` file and `dbt_project.yml` file: From 999f5ada89247c5e49d04badb55d658e3f5f57e8 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 24 Dec 2024 11:05:59 +0000 Subject: [PATCH 36/81] Update website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md --- website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md b/website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md index fabd2073995..19aeccf3224 100644 --- a/website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md +++ b/website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md @@ -38,7 +38,7 @@ Whether you use a Git integration that natively connects with dbt Cloud or prefe icon="dbt-bit"/> From 67ec2a07acc0e1d805ef5640b105dd1998b2981a Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 24 Dec 2024 11:07:58 +0000 Subject: [PATCH 37/81] Update managed-repository.md --- website/docs/docs/cloud/git/managed-repository.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/website/docs/docs/cloud/git/managed-repository.md b/website/docs/docs/cloud/git/managed-repository.md index ef10d6324a7..cce3c8ac9f0 100644 --- a/website/docs/docs/cloud/git/managed-repository.md +++ b/website/docs/docs/cloud/git/managed-repository.md @@ -6,7 +6,7 @@ pagination_next: "docs/cloud/git/import-a-project-by-git-url" pagination_prev: "docs/cloud/git/git-configuration-in-dbt-cloud" --- -If you do not already have a git repository for your dbt project, you can let dbt Cloud manage a repository for you. Managed repositories are a great way to trial dbt without needing to create a new repository. +If you don't already have a Git repository for your dbt project, you can let dbt Cloud manage a repository for you. Managed repositories are a great way to trial dbt without needing to create a new repository. :::info We do not recommend using a managed repository in a production environment. You will not be able to use git features like pull requests which are part of our recommended version control best practices. @@ -14,7 +14,7 @@ We do not recommend using a managed repository in a production environment. You To set up a project with a managed repository: -1. From your Account settings in dbt Cloud, select the project you want to set up with a managed repository. If the project already has a repository set up, you need to edit the repository settings and disconnect the existing repository. +1. From your A**ccount settings** in dbt Cloud, select the project you want to set up with a managed repository. If the project already has a repository set up, you need to edit the repository settings and disconnect the existing repository. 2. Click **Edit** for the project. 3. Under Repository, click **Configure repository**. 4. Select **Managed**. @@ -22,4 +22,6 @@ To set up a project with a managed repository: 6. Click **Create**. -dbt Cloud will host and manage this repository for you. If in the future you choose to host this repository elsewhere, you can export the information from dbt Cloud at any time. Refer to [Move from a managed repository to a self-hosted repository](/faqs/Git/managed-repo) for more information. +- dbt Cloud will host and manage this repository for you. +- If in the future you choose to host this repository elsewhere, you can export the information from dbt Cloud at any time. + - Refer to [Move from a managed repository to a self-hosted repository](/faqs/Git/managed-repo) for more information. From f98a1a70fb5f4947f5cdd177572381c459155457 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Tue, 24 Dec 2024 12:21:45 +0000 Subject: [PATCH 38/81] natalies feedback --- .../docs/reference/resource-configs/tags.md | 31 ++++++++++--------- 1 file changed, 17 insertions(+), 14 deletions(-) diff --git a/website/docs/reference/resource-configs/tags.md b/website/docs/reference/resource-configs/tags.md index 27be1f71766..b77c1472032 100644 --- a/website/docs/reference/resource-configs/tags.md +++ b/website/docs/reference/resource-configs/tags.md @@ -164,14 +164,14 @@ select ... -Then, run part of your project like so: +Run resources with specific tags (or exclude resources with specific tags) with the following commands: -``` +```shell # Run all models tagged "daily" -$ dbt run --select tag:daily + dbt run --select tag:daily # Run all models tagged "daily", except those that are tagged hourly -$ dbt run --select tag:daily --exclude tag:hourly + dbt run --select tag:daily --exclude tag:hourly ``` ### Apply tags to seeds @@ -204,7 +204,10 @@ seeds: +:::tip Upgrade to dbt Core 1.9 + Applying tags to saved queries is only available in dbt Core versions 1.9 and later. +::: @@ -223,11 +226,11 @@ This following example shows how to apply a tag to a saved query in the `dbt_pro -Then, run part of your project like so: +Then run resources with a specific tag with the following commands: -``` +```shell # Run all resources tagged "order_metrics" -dbt run --select tag:order_metrics + dbt run --select tag:order_metrics ``` The second example shows how to apply multiple tags to a saved query in the `semantic_model.yml` file. The saved query is then tagged with `order_metrics` and `hourly`. @@ -247,11 +250,11 @@ saved_queries: -Then, run part of your project like so: +Run resources with multiple tags with the following commands: -``` +```shell # Run all resources tagged "order_metrics" and "hourly" -dbt build --select tag:order_metrics tag:hourly + dbt build --select tag:order_metrics tag:hourly ``` @@ -304,8 +307,8 @@ sources: In the example above, the `unique` test would be selected by any of these four tags: ```bash -$ dbt test --select tag:top_level -$ dbt test --select tag:table_level -$ dbt test --select tag:column_level -$ dbt test --select tag:test_level +dbt test --select tag:top_level +dbt test --select tag:table_level +dbt test --select tag:column_level +dbt test --select tag:test_level ``` From 642912c84934c807d3cbaf19f077094b48fcecaf Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Tue, 24 Dec 2024 12:22:57 +0000 Subject: [PATCH 39/81] grammar fix --- website/docs/reference/resource-configs/tags.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/docs/reference/resource-configs/tags.md b/website/docs/reference/resource-configs/tags.md index b77c1472032..c222df8c1ae 100644 --- a/website/docs/reference/resource-configs/tags.md +++ b/website/docs/reference/resource-configs/tags.md @@ -164,7 +164,7 @@ select ... -Run resources with specific tags (or exclude resources with specific tags) with the following commands: +Run resources with specific tags (or exclude resources with specific tags) using the following commands: ```shell # Run all models tagged "daily" @@ -226,7 +226,7 @@ This following example shows how to apply a tag to a saved query in the `dbt_pro -Then run resources with a specific tag with the following commands: +Then run resources with a specific tag using the following commands: ```shell # Run all resources tagged "order_metrics" @@ -250,7 +250,7 @@ saved_queries: -Run resources with multiple tags with the following commands: +Run resources with multiple tags using the following commands: ```shell # Run all resources tagged "order_metrics" and "hourly" From b17bec261ef43ec32200f0fef51b389c389a2392 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 24 Dec 2024 14:14:25 +0000 Subject: [PATCH 40/81] Update website/docs/faqs/Git/gitignore.md Co-authored-by: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> --- website/docs/faqs/Git/gitignore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/faqs/Git/gitignore.md b/website/docs/faqs/Git/gitignore.md index 350fd1e77c9..1b9013a4473 100644 --- a/website/docs/faqs/Git/gitignore.md +++ b/website/docs/faqs/Git/gitignore.md @@ -55,7 +55,7 @@ For more info on `gitignore` syntax, refer to the [Git docs](https://git-scm.com - + Sometimes it's necessary to use the git providers web interface to fix a broken `.gitignore` file. Although the specific steps may vary across providers, the general process remains the same. From 366ad9434ca4ba02cc5db0efdefa09880626e36b Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 24 Dec 2024 14:14:51 +0000 Subject: [PATCH 41/81] Update website/docs/docs/cloud/git/managed-repository.md Co-authored-by: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> --- website/docs/docs/cloud/git/managed-repository.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/git/managed-repository.md b/website/docs/docs/cloud/git/managed-repository.md index cce3c8ac9f0..eb782839614 100644 --- a/website/docs/docs/cloud/git/managed-repository.md +++ b/website/docs/docs/cloud/git/managed-repository.md @@ -9,7 +9,7 @@ pagination_prev: "docs/cloud/git/git-configuration-in-dbt-cloud" If you don't already have a Git repository for your dbt project, you can let dbt Cloud manage a repository for you. Managed repositories are a great way to trial dbt without needing to create a new repository. :::info -We do not recommend using a managed repository in a production environment. You will not be able to use git features like pull requests which are part of our recommended version control best practices. +dbt Labs recommends against using a managed repository in a production environment. You can't use Git features like pull requests, which are part of our recommended version control best practices. ::: To set up a project with a managed repository: From 436bdbc7eee226917bfa91eb7294f88ef7a7175d Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 24 Dec 2024 14:15:06 +0000 Subject: [PATCH 42/81] Update website/docs/docs/cloud/git/managed-repository.md Co-authored-by: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> --- website/docs/docs/cloud/git/managed-repository.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/git/managed-repository.md b/website/docs/docs/cloud/git/managed-repository.md index eb782839614..f26d5d475ac 100644 --- a/website/docs/docs/cloud/git/managed-repository.md +++ b/website/docs/docs/cloud/git/managed-repository.md @@ -14,7 +14,7 @@ dbt Labs recommends against using a managed repository in a production environme To set up a project with a managed repository: -1. From your A**ccount settings** in dbt Cloud, select the project you want to set up with a managed repository. If the project already has a repository set up, you need to edit the repository settings and disconnect the existing repository. +1. From your **Account settings** in dbt Cloud, select the project you want to set up with a managed repository. If the project already has a repository set up, you need to edit the repository settings and disconnect the existing repository. 2. Click **Edit** for the project. 3. Under Repository, click **Configure repository**. 4. Select **Managed**. From bbe8d80bf36bd26f15e0494271dba9d883c2a98b Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 24 Dec 2024 14:18:58 +0000 Subject: [PATCH 43/81] Update website/docs/docs/cloud/git/managed-repository.md Co-authored-by: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> --- website/docs/docs/cloud/git/managed-repository.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/git/managed-repository.md b/website/docs/docs/cloud/git/managed-repository.md index f26d5d475ac..67d957fcee6 100644 --- a/website/docs/docs/cloud/git/managed-repository.md +++ b/website/docs/docs/cloud/git/managed-repository.md @@ -22,6 +22,6 @@ To set up a project with a managed repository: 6. Click **Create**. -- dbt Cloud will host and manage this repository for you. +- dbt Cloud hosts and manages this repository for you. - If in the future you choose to host this repository elsewhere, you can export the information from dbt Cloud at any time. - Refer to [Move from a managed repository to a self-hosted repository](/faqs/Git/managed-repo) for more information. From ed0a93a531bb25e91bca2909a0ad6d22c720831d Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 24 Dec 2024 14:22:05 +0000 Subject: [PATCH 44/81] Update git-configuration-in-dbt-cloud.md --- website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md b/website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md index 19aeccf3224..32e1581cc94 100644 --- a/website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md +++ b/website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md @@ -5,7 +5,7 @@ hide_table_of_contents: true pagination_next: "docs/cloud/git/managed-repository" --- -[Version control](/docs/collaborate/git/version-control-basics) is an essential part of the dbt workflow, enabling teams to collaborate effectively and maintain a history of changes to their dbt projects. +[Version control](/docs/collaborate/git/version-control-basics) — a system allows you and your teammates to work collaboratively, safely, and simultaneously on a single project — is an essential part of the dbt workflow, enabling teams to collaborate effectively and maintain a history of changes to their dbt projects. In dbt Cloud, you can configure Git integrations to manage your dbt project code with ease. dbt Cloud offers multiple ways to integrate with you Git provider, catering to diverse team needs and preferences. From 066a3e235790f1872e94587929081c2a47ca1d07 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 24 Dec 2024 14:22:40 +0000 Subject: [PATCH 45/81] Update website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md --- website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md b/website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md index 32e1581cc94..ea4b23a92ef 100644 --- a/website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md +++ b/website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md @@ -5,7 +5,7 @@ hide_table_of_contents: true pagination_next: "docs/cloud/git/managed-repository" --- -[Version control](/docs/collaborate/git/version-control-basics) — a system allows you and your teammates to work collaboratively, safely, and simultaneously on a single project — is an essential part of the dbt workflow, enabling teams to collaborate effectively and maintain a history of changes to their dbt projects. +[Version control](/docs/collaborate/git/version-control-basics) — a system allows you and your teammates to work safely and simultaneously on a single project — is an essential part of the dbt workflow. It enables teams to collaborate effectively and maintain a history of changes to their dbt projects. In dbt Cloud, you can configure Git integrations to manage your dbt project code with ease. dbt Cloud offers multiple ways to integrate with you Git provider, catering to diverse team needs and preferences. From 8f9f3de8764123e335830c17356bcefc428c027b Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 24 Dec 2024 14:29:18 +0000 Subject: [PATCH 46/81] Update managed-repository.md --- website/docs/docs/cloud/git/managed-repository.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/website/docs/docs/cloud/git/managed-repository.md b/website/docs/docs/cloud/git/managed-repository.md index 67d957fcee6..64379611825 100644 --- a/website/docs/docs/cloud/git/managed-repository.md +++ b/website/docs/docs/cloud/git/managed-repository.md @@ -6,7 +6,10 @@ pagination_next: "docs/cloud/git/import-a-project-by-git-url" pagination_prev: "docs/cloud/git/git-configuration-in-dbt-cloud" --- -If you don't already have a Git repository for your dbt project, you can let dbt Cloud manage a repository for you. Managed repositories are a great way to trial dbt without needing to create a new repository. +Managed repositories are a great way to trial dbt without needing to create a new repository. If you don't already have a Git repository for your dbt project, you can let dbt Cloud host and manage a repository for you. + +If in the future you choose to host this repository elsewhere, you can export the information from dbt Cloud at any time. Refer to [Move from a managed repository to a self-hosted repository](/faqs/Git/managed-repo) for more information on how to do that. + :::info dbt Labs recommends against using a managed repository in a production environment. You can't use Git features like pull requests, which are part of our recommended version control best practices. @@ -22,6 +25,3 @@ To set up a project with a managed repository: 6. Click **Create**. -- dbt Cloud hosts and manages this repository for you. -- If in the future you choose to host this repository elsewhere, you can export the information from dbt Cloud at any time. - - Refer to [Move from a managed repository to a self-hosted repository](/faqs/Git/managed-repo) for more information. From 62a643950ad9dbfce504c5367b0e29a4c7ea9d05 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 24 Dec 2024 14:32:48 +0000 Subject: [PATCH 47/81] Update website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md --- website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md b/website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md index ea4b23a92ef..57558f7cb5b 100644 --- a/website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md +++ b/website/docs/docs/cloud/git/git-configuration-in-dbt-cloud.md @@ -5,7 +5,7 @@ hide_table_of_contents: true pagination_next: "docs/cloud/git/managed-repository" --- -[Version control](/docs/collaborate/git/version-control-basics) — a system allows you and your teammates to work safely and simultaneously on a single project — is an essential part of the dbt workflow. It enables teams to collaborate effectively and maintain a history of changes to their dbt projects. +[Version control](/docs/collaborate/git/version-control-basics) — a system that allows you and your teammates to work safely and simultaneously on a single project — is an essential part of the dbt workflow. It enables teams to collaborate effectively and maintain a history of changes to their dbt projects. In dbt Cloud, you can configure Git integrations to manage your dbt project code with ease. dbt Cloud offers multiple ways to integrate with you Git provider, catering to diverse team needs and preferences. From 96225898e64a14926e104a54bacc887cbda9543b Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 24 Dec 2024 22:24:16 +0000 Subject: [PATCH 48/81] clarify ado behavior (#6705) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit this pr clarifies that for ado, private packages have to be configured using the `org/repo` path. more details in internal slack: https://dbt-labs.slack.com/archives/C07C793M6EB/p1734715084356599?thread_ts=1734359013.848149&cid=C07C793M6EB --- 🚀 Deployment available! Here are the direct links to the updated files: - https://docs-getdbt-com-git-mirnawong1-patch-28-dbt-labs.vercel.app/docs/build/packages --------- Co-authored-by: Leona B. Campbell <3880403+runleonarun@users.noreply.github.com> --- website/docs/docs/build/packages.md | 35 +++++++++++++++++++++-------- 1 file changed, 26 insertions(+), 9 deletions(-) diff --git a/website/docs/docs/build/packages.md b/website/docs/docs/build/packages.md index 7a2c08d3e70..5ee619989bc 100644 --- a/website/docs/docs/build/packages.md +++ b/website/docs/docs/build/packages.md @@ -165,27 +165,44 @@ dbt Cloud supports private packages from [supported](#prerequisites) Git repos l #### Prerequisites -To use native private packages, you must have one of the following Git providers configured in the **Integrations** section of your **Account settings**: -- [GitHub](/docs/cloud/git/connect-github) -- [Azure DevOps](/docs/cloud/git/connect-azure-devops) -- Support for GitLab is coming soon. - +- To use native private packages, you must have one of the following Git providers configured in the **Integrations** section of your **Account settings**: + - [GitHub](/docs/cloud/git/connect-github) + - [Azure DevOps](/docs/cloud/git/connect-azure-devops) + - Private packages only work within a single Azure DevOps project. If your repositories are in different projects within the same organization, you can't reference them in the `private` key at this time. + - For Azure DevOps, use the `org/repo` path (not the `org_name/project_name/repo_name` path) with the project tier inherited from the integrated source repository. + - Support for GitLab is coming soon. #### Configuration -Use the `private` key in your `packages.yml` or `dependencies.yml` to clone package repos using your existing dbt Cloud Git integration without having to provision an access token or create a dbt Cloud environment variable: +Use the `private` key in your `packages.yml` or `dependencies.yml` to clone package repos using your existing dbt Cloud Git integration without having to provision an access token or create a dbt Cloud environment variable. + ```yaml packages: - - private: dbt-labs/awesome_repo + - private: dbt-labs/awesome_repo # your-org/your-repo path - package: normal packages - - [...] + [...] ``` + + +:::tip Azure DevOps considerations + +- Private packages currently only work if the package repository is in the same Azure DevOps project as the source repo. +- Use the `org/repo` path (not the normal ADO `org_name/project_name/repo_name` path) in the `private` key. +- Repositories in different Azure DevOps projects is currently not supported until a future update. +You can use private packages by specifying `org/repo` in the `private` key: + + + +```yaml +packages: + - private: my-org/my-repo # Works if your ADO source repo and package repo are in the same project +``` +::: You can pin private packages similar to regular dbt packages: From 65a46f1d8e0e5909829080c7b9429ef1c7f54c59 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Tue, 24 Dec 2024 22:32:39 +0000 Subject: [PATCH 49/81] add clarifying info on how MF joins works (#6690) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit this pr adds more detail about the joins supported by mf. currently it mentions left joins but there's no info on outer joins or semantic valiations. this pr adds more info on the types of joins available, when they're used, semantic validations available, and practical sql examples raised by [internal slack](https://dbt-labs.slack.com/archives/C03KHQRQUBX/p1733947531180079) Resolves #6650 --- 🚀 Deployment available! Here are the direct links to the updated files: - https://docs-getdbt-com-git-update-mf-joins-dbt-labs.vercel.app/docs/build/join-logic --------- Co-authored-by: Leona B. Campbell <3880403+runleonarun@users.noreply.github.com> --- website/docs/docs/build/join-logic.md | 82 +++++++++++++++++++++++---- 1 file changed, 70 insertions(+), 12 deletions(-) diff --git a/website/docs/docs/build/join-logic.md b/website/docs/docs/build/join-logic.md index 99d63b38657..d11c0248d25 100644 --- a/website/docs/docs/build/join-logic.md +++ b/website/docs/docs/build/join-logic.md @@ -10,24 +10,28 @@ Joins are a powerful part of MetricFlow and simplify the process of making all v Joins use `entities` defined in your semantic model configs as the join keys between tables. Assuming entities are defined in the semantic model, MetricFlow creates a graph using the semantic models as nodes and the join paths as edges to perform joins automatically. MetricFlow chooses the appropriate join type and avoids fan-out or chasm joins with other tables based on the entity types. -
- What are fan-out or chasm joins? -
-
— Fan-out joins are when one row in a table is joined to multiple rows in another table, resulting in more output rows than input rows.

- — Chasm joins are when two tables have a many-to-many relationship through an intermediate table, and the join results in duplicate or missing data.
-
-
- + +- Fan-out joins are when one row in a table is joined to multiple rows in another table, resulting in more output rows than input rows. +- Chasm joins are when two tables have a many-to-many relationship through an intermediate table, and the join results in duplicate or missing data. + ## Types of joins :::tip Joins are auto-generated MetricFlow automatically generates the necessary joins to the defined semantic objects, eliminating the need for you to create new semantic models or configuration files. -This document explains the different types of joins that can be used with entities and how to query them using the CLI. +This section explains the different types of joins that can be used with entities and how to query them. ::: -MetricFlow primarily uses left joins for joins, and restricts the use of fan-out and chasm joins. Refer to the table below to identify which joins are or aren't allowed based on specific entity types to prevent the creation of risky joins. +Metricflow uses these specific join strategies: + +- Primarily uses left joins when joining `fct` and `dim` models. Left joins make sure all rows from the "base" table are retained, while matching rows are included from the joined table. +- For queries that involve multiple `fct` models, MetricFlow uses full outer joins to ensure all data points are captured, even when some `dim` or `fct` models are missing in certain tables. +- MetricFlow restricts the use of fan-out and chasm joins. + +Refer to [SQL examples](#sql-examples) for more information on how MetricFlow handles joins in practice. + +The following table identifies which joins are allowed based on specific entity types to prevent the creation of risky joins. This table primarily represents left joins unless otherwise specified. For scenarios involving multiple `fct` models, MetricFlow uses full outer joins. | entity type - Table A | entity type - Table B | Join type | |---------------------------|---------------------------|----------------------| @@ -39,9 +43,19 @@ MetricFlow primarily uses left joins for joins, and restricts the use of fan-out | Unique | Foreign | ❌ Fan-out (Not allowed) | | Foreign | Primary | âś… Left | | Foreign | Unique | âś… Left | -| Foreign | Foreign | ❌ Fan-out (Not allowed) | +| Foreign | Foreign | ❌ Fan-out (Not allowed) | + +### Semantic validation -### Example +MetricFlow performs semantic validation by executing `explain` queries in the data platform to ensure that the generated SQL gets executed without errors. This validation includes: + +- Verifying that all referenced tables and columns exist. +- Ensuring the data platform supports SQL functions, such as `date_diff(x, y)`. +- Checking for ambiguous joins or paths in multi-hop joins. + +If validation fails, MetricFlow surfaces errors for users to address before executing the query. + +## Example The following example uses two semantic models with a common entity and shows a MetricFlow query that requires a join between the two semantic models. The two semantic models are: - `transactions` @@ -83,6 +97,50 @@ dbt sl query --metrics average_purchase_price --group-by metric_time,user_id__ty mf query --metrics average_purchase_price --group-by metric_time,user_id__type # In dbt Core ``` +#### SQL examples + +These SQL examples show how MetricFlow handles both left join and full outer join scenarios in practice: + + + + +Using the previous example for `transactions` and `user_signup` semantic models, this shows a left join between those two semantic models. + +```sql +select + transactions.user_id, + transactions.purchase_price, + user_signup.type +from transactions +left outer join user_signup + on transactions.user_id = user_signup.user_id +where transactions.purchase_price is not null +group by + transactions.user_id, + user_signup.type; +``` + + + + +If you have multiple `fct` models, let's say `sales` and `returns`, MetricFlow uses full outer joins to ensure all data points are captured. + +This example shows a full outer join between the `sales` and `returns` semantic models. + +```sql +select + sales.user_id, + sales.total_sales, + returns.total_returns +from sales +full outer join returns + on sales.user_id = returns.user_id +where sales.user_id is not null or returns.user_id is not null; +``` + + + + ## Multi-hop joins MetricFlow allows users to join measures and dimensions across a graph of entities by moving from one table to another within a graph. This is referred to as "multi-hop join". From 7cc790196e5974f3436db818e52bd63efe2bb288 Mon Sep 17 00:00:00 2001 From: numb86 <16703337+numb86@users.noreply.github.com> Date: Wed, 25 Dec 2024 23:14:58 +0900 Subject: [PATCH 50/81] Fix fragment identifier --- website/docs/reference/commands/run-operation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/commands/run-operation.md b/website/docs/reference/commands/run-operation.md index 6b842b3064c..35dab07874f 100644 --- a/website/docs/reference/commands/run-operation.md +++ b/website/docs/reference/commands/run-operation.md @@ -7,7 +7,7 @@ id: "run-operation" ### Overview -The `dbt run-operation` command is used to invoke a macro. For usage information, consult the docs on [operations](/docs/build/hooks-operations#operations). +The `dbt run-operation` command is used to invoke a macro. For usage information, consult the docs on [operations](/docs/build/hooks-operations#about-operations). ### Usage ``` From 6e609e6ce095b381b53b399f152bb1c902cdd1b9 Mon Sep 17 00:00:00 2001 From: Anks S <6273915+asarraf@users.noreply.github.com> Date: Mon, 30 Dec 2024 11:17:01 -0800 Subject: [PATCH 51/81] Update 08-upgrading-to-v1.7.md Corrections related to dbt-clean --- .../docs/dbt-versions/core-upgrade/08-upgrading-to-v1.7.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/docs/dbt-versions/core-upgrade/08-upgrading-to-v1.7.md b/website/docs/docs/dbt-versions/core-upgrade/08-upgrading-to-v1.7.md index b98a76295cf..0846d89516b 100644 --- a/website/docs/docs/dbt-versions/core-upgrade/08-upgrading-to-v1.7.md +++ b/website/docs/docs/dbt-versions/core-upgrade/08-upgrading-to-v1.7.md @@ -62,11 +62,11 @@ dbt Core v1.5 introduced model governance which we're continuing to refine. v1. - **[Breaking change detection](/reference/resource-properties/versions#detecting-breaking-changes) for models with contracts enforced:** When dbt detects a breaking change to a model with an enforced contract during state comparison, it will now raise an error for versioned models and a warning for models that are not versioned. - **[Set `access` as a config](/reference/resource-configs/access):** You can now set a model's `access` within config blocks in the model's file or in the `dbt_project.yml` for an entire subfolder at once. - **[Type aliasing for model contracts](/reference/resource-configs/contract):** dbt will use each adapter's built-in type aliasing for user-provided data types—meaning you can now write `string` always, and dbt will translate to `text` on Postgres/Redshift. This is "on" by default, but you can opt-out. -- **[Raise warning for numeric types](/reference/resource-configs/contract):** Because of issues when putting `numeric` in model contracts without considering that default values such as `numeric(38,0)` might round decimals accordingly. dbt will now warn you if it finds a numeric type without specified precision/scale. +- **[Raise warning for numeric types](/reference/resource-configs/contract):** Because of issues when putting `numeric` in model contracts without considering that default values such as `numeric(38,0)` might round decimals accordingly. dbt will now warn you if it finds a numeric type without specified precision/scale. ### dbt clean -Starting in v1.7, `dbt clean` will only clean paths within the current working directory. The `--no-clean-project-files-only` flag will delete all paths specified in the `clean-targets` section of `dbt_project.yml`, even if they're outside the dbt project. +Starting in v1.7, [dbt clean](/reference/commands/clean) will only clean paths within the current working directory. The `--no-clean-project-files-only` flag will delete all paths specified in the `clean-targets` section of `dbt_project.yml`, even if they're outside the dbt project. Supported flags: - `--clean-project-files-only` (default) From 05c1fa2a730d94785f5fa4b4e2adbe19d164a7d2 Mon Sep 17 00:00:00 2001 From: Anks S <6273915+asarraf@users.noreply.github.com> Date: Mon, 30 Dec 2024 11:17:56 -0800 Subject: [PATCH 52/81] Update clean.md Refined documentation for dot-clean --- website/docs/reference/commands/clean.md | 29 +++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/website/docs/reference/commands/clean.md b/website/docs/reference/commands/clean.md index 23a3f6080ce..d306d82446c 100644 --- a/website/docs/reference/commands/clean.md +++ b/website/docs/reference/commands/clean.md @@ -4,6 +4,33 @@ sidebar_label: "clean" id: "clean" --- -`dbt clean` is a utility function that deletes all folders specified in the [`clean-targets`](/reference/project-configs/clean-targets) list specified in `dbt_project.yml`. You can use this to delete the `dbt_packages` and `target` directories. +`dbt clean` is a utility function that deletes the paths specified within the [`clean-targets`](/reference/project-configs/clean-targets) list in the `dbt_project.yml` file. It helps by removing unnecessary files or directories generated during the execution of other dbt commands, ensuring a clean state for the project. +## Example usage +``` +dbt clean +``` + +## Supported flags +### --clean-project-files-only (default) +Deletes all the paths within the project directory specified in `clean-targets`. + +:::note +Avoid using paths outside the dbt project; otherwise, you will see an error. +::: + + +#### Example usage +``` +$ dbt clean --clean-project-files-only +``` + +### --no-clean-project-files-only +Deletes all the paths specified in the `clean-targets` list of `dbt_project.yml`, including those outside the current dbt project. + +``` +$ dbt clean --no-clean-project-files-only +``` + +## dbt clean with remote file system To avoid complex permissions issues and potentially deleting crucial aspects of the remote file system without access to fix them, this command does not work when interfacing with the RPC server that powers the dbt Cloud IDE. Instead, when working in dbt Cloud, the `dbt deps` command cleans before it installs packages automatically. The `target` folder can be manually deleted from the sidebar file tree if needed. From 7cb490c125993c25b668331590399d449d28b41a Mon Sep 17 00:00:00 2001 From: czasoprzestrzenny <46657059+czasoprzestrzenny@users.noreply.github.com> Date: Thu, 2 Jan 2025 18:52:09 +0100 Subject: [PATCH 53/81] Update incremental-microbatch.md --- website/docs/docs/build/incremental-microbatch.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/incremental-microbatch.md b/website/docs/docs/build/incremental-microbatch.md index 77f9ac274b0..9d51b77e2e4 100644 --- a/website/docs/docs/build/incremental-microbatch.md +++ b/website/docs/docs/build/incremental-microbatch.md @@ -389,7 +389,7 @@ models: incremental_strategy='microbatch', event_time='session_start', begin='2020-01-01', - batch_size='day + batch_size='day', concurrent_batches=true, # value set to true to run batches in parallel ... ) From b127ed336516697080d20f89659fb8c38da10d75 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 6 Jan 2025 10:25:03 +0000 Subject: [PATCH 54/81] Update batch_size.md update sql example to align with `batch_size` config. previously the example only showed how to configure the `lookback` config, which isn't related to this page. --- website/docs/reference/resource-configs/batch_size.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/website/docs/reference/resource-configs/batch_size.md b/website/docs/reference/resource-configs/batch_size.md index 0110da53bb2..6237a522f98 100644 --- a/website/docs/reference/resource-configs/batch_size.md +++ b/website/docs/reference/resource-configs/batch_size.md @@ -48,7 +48,8 @@ Example in sql model config block: ```sql {{ config( - lookback='day + materialized='incremental', + batch_size='day' ) }} ``` From 491ffb2926bcd6d9f7f8d4d323c6d05ed35ba23c Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 6 Jan 2025 12:21:50 +0000 Subject: [PATCH 55/81] add clarification about groups and licenses adding note about permissions availability and plan per convo in [internal slack](https://dbt-labs.slack.com/archives/C017GDLAF7D/p1736161718479829?thread_ts=1736159296.110149&cid=C017GDLAF7D). also adding separate header to clarify new groups is for dbt enterprise. --- .../docs/cloud/manage-access/about-access.md | 26 +++++++++++-------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/website/docs/docs/cloud/manage-access/about-access.md b/website/docs/docs/cloud/manage-access/about-access.md index b9d23b28add..f94457379ca 100644 --- a/website/docs/docs/cloud/manage-access/about-access.md +++ b/website/docs/docs/cloud/manage-access/about-access.md @@ -8,12 +8,10 @@ pagination_prev: null :::info "User access" is not "Model access" -This page is specific to user groups and access, which includes: -- User licenses, permissions, and group memberships -- Role-based access controls for projects and environments -- Single sign-on and secure authentication +This page covers user groups and access, including: +- User licenses, permissions, group memberships, role-based access controls for projects and environments, single sign-on, and secure authentication -"Model groups and access" is a feature specific to models and their availability across projects. Refer to [Model access](/docs/collaborate/govern/model-access) for more info on what it means for your dbt projects. +For model-specific access and their availability across projects, refer to [Model access](/docs/collaborate/govern/model-access). ::: @@ -21,7 +19,7 @@ This page is specific to user groups and access, which includes: You can regulate access to dbt Cloud by various measures, including licenses, groups, permissions, and role-based access control (RBAC). To understand the possible approaches to user access to dbt Cloud features and functionality, you should first know how we approach users and groups. -### Users +## Users Individual users in dbt Cloud can be people you [manually invite](/docs/cloud/manage-access/invite-users) or grant access via an external identity provider (IdP), such as Microsoft Entra ID, Okta, or Google Workspace. @@ -33,10 +31,13 @@ You can edit an existing user's license by navigating to the **Users** section o +## Groups -### Groups +Groups in dbt Cloud serve much of the same purpose as they do in traditional directory tools — to gather individual users together to make bulk assignment of permissions easier. Admins use groups in dbt Cloud to assign [licenses](#licenses) and [permissions](#permissions). -Groups in dbt Cloud serve much of the same purpose as they do in traditional directory tools — to gather individual users together to make bulk assignment of permissions easier. Admins use groups in dbt Cloud to assign [licenses](#licenses) and [permissions](#permissions). The permissions are more granular than licenses, and you only assign them at the group level; _you can’t assign permissions at the user level._ Every user in dbt Cloud must be assigned to at least one group. +- The permissions are more granular than licenses, and you only assign them at the group level; _you can’t assign permissions at the user level._ +- The permissions available depends on whether you're on an [Enterprise](/docs/cloud/manage-access/enterprise-permissions) or [self-service Team](/docs/cloud/manage-access/self-service-permissions) plan. +- Every user in dbt Cloud must be assigned to at least one group. There are three default groups available as soon as you create your dbt Cloud account (the person who created the account is added to all three automatically): @@ -46,7 +47,10 @@ There are three default groups available as soon as you create your dbt Cloud ac We recommend deleting the default `Owner`, `Member`, and `Everyone` groups before deploying and replacing them with your organizational groups. This prevents users from receiving more elevated privileges than they should and helps admins ensure they are properly placed. -Create new groups from the **Groups & Licenses** section of the **Account settings**. If you use an external IdP for SSO, you can sync those SSO groups to dbt Cloud from the **Group details** pane when creating or editing existing groups. +### Create new groups + +- Create new groups from the **Groups & Licenses** section of the **Account settings**. +- If you use an external IdP for SSO, you can sync those SSO groups to dbt Cloud from the **Group details** pane when creating or editing existing groups. @@ -56,7 +60,7 @@ If a user is assigned licenses and permissions from multiple groups, the group t ::: -#### SSO mappings +### SSO mappings SSO Mappings connect an identity provider (IdP) group membership to a dbt Cloud group. When users log into dbt Cloud via a supported identity provider, their IdP group memberships sync with dbt Cloud. Upon logging in successfully, the user's group memberships (and permissions) will automatically adjust within dbt Cloud. @@ -215,4 +219,4 @@ Each dbt Cloud plan has a base number of Developer and Read-Only licenses. You c For detailed steps, refer to [Users and licenses](/docs/cloud/manage-access/seats-and-users#licenses). - \ No newline at end of file + From 8300f42200f7d20b8e53770454cb184ad888b462 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 6 Jan 2025 12:26:45 +0000 Subject: [PATCH 56/81] Update about-access.md --- website/docs/docs/cloud/manage-access/about-access.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/docs/cloud/manage-access/about-access.md b/website/docs/docs/cloud/manage-access/about-access.md index f94457379ca..537494f8c02 100644 --- a/website/docs/docs/cloud/manage-access/about-access.md +++ b/website/docs/docs/cloud/manage-access/about-access.md @@ -33,10 +33,10 @@ You can edit an existing user's license by navigating to the **Users** section o ## Groups -Groups in dbt Cloud serve much of the same purpose as they do in traditional directory tools — to gather individual users together to make bulk assignment of permissions easier. Admins use groups in dbt Cloud to assign [licenses](#licenses) and [permissions](#permissions). +Groups in dbt Cloud serve much of the same purpose as they do in traditional directory tools — to gather individual users together to make bulk assignment of permissions easier. The permissions available depends on whether you're on an [Enterprise](/docs/cloud/manage-access/enterprise-permissions) or [self-service Team](/docs/cloud/manage-access/self-service-permissions) plan. +- Admins use groups in dbt Cloud to assign [licenses](#licenses) and [permissions](#permissions). - The permissions are more granular than licenses, and you only assign them at the group level; _you can’t assign permissions at the user level._ -- The permissions available depends on whether you're on an [Enterprise](/docs/cloud/manage-access/enterprise-permissions) or [self-service Team](/docs/cloud/manage-access/self-service-permissions) plan. - Every user in dbt Cloud must be assigned to at least one group. There are three default groups available as soon as you create your dbt Cloud account (the person who created the account is added to all three automatically): From d29986fe1cf0cdd2b678a37f92537d9060903bbf Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 6 Jan 2025 12:39:37 +0000 Subject: [PATCH 57/81] Update website/docs/reference/commands/clean.md --- website/docs/reference/commands/clean.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/website/docs/reference/commands/clean.md b/website/docs/reference/commands/clean.md index d306d82446c..3c93be18250 100644 --- a/website/docs/reference/commands/clean.md +++ b/website/docs/reference/commands/clean.md @@ -12,6 +12,10 @@ dbt clean ``` ## Supported flags +To view the supported flags for the `dbt clean` command in the terminal, use the `--help` flag like this: + +```shell +dbt clean --help ### --clean-project-files-only (default) Deletes all the paths within the project directory specified in `clean-targets`. From b152d3fbbba4acbe344bd8b94270c8a8587f6253 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 6 Jan 2025 12:40:19 +0000 Subject: [PATCH 58/81] Update clean.md --- website/docs/reference/commands/clean.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/website/docs/reference/commands/clean.md b/website/docs/reference/commands/clean.md index 3c93be18250..117dd0585c4 100644 --- a/website/docs/reference/commands/clean.md +++ b/website/docs/reference/commands/clean.md @@ -16,6 +16,10 @@ To view the supported flags for the `dbt clean` command in the terminal, use the ```shell dbt clean --help +``` + +This command displays detailed information about the available flags you can use, including its description and usage. + ### --clean-project-files-only (default) Deletes all the paths within the project directory specified in `clean-targets`. From bb87c302c7a302cfcf2debc627264773979eea3e Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 6 Jan 2025 12:53:07 +0000 Subject: [PATCH 59/81] Update clean.md small tweaks --- website/docs/reference/commands/clean.md | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/website/docs/reference/commands/clean.md b/website/docs/reference/commands/clean.md index 117dd0585c4..24c8ffba5b4 100644 --- a/website/docs/reference/commands/clean.md +++ b/website/docs/reference/commands/clean.md @@ -12,16 +12,20 @@ dbt clean ``` ## Supported flags -To view the supported flags for the `dbt clean` command in the terminal, use the `--help` flag like this: + +This section will briefly explain the following flags: + +- `--clean-project-files-only` (default) +- `--no-clean-project-files-only` + +To view the list of all supported flags for the `dbt clean` command in the terminal, use the `--help` flag, which will display detailed information about the available flags you can use, including its description and usage: ```shell dbt clean --help ``` -This command displays detailed information about the available flags you can use, including its description and usage. - ### --clean-project-files-only (default) -Deletes all the paths within the project directory specified in `clean-targets`. +By default, dbt deletes all the paths within the project directory specified in `clean-targets`. :::note Avoid using paths outside the dbt project; otherwise, you will see an error. @@ -29,15 +33,15 @@ Avoid using paths outside the dbt project; otherwise, you will see an error. #### Example usage -``` -$ dbt clean --clean-project-files-only +```shell +dbt clean --clean-project-files-only ``` ### --no-clean-project-files-only Deletes all the paths specified in the `clean-targets` list of `dbt_project.yml`, including those outside the current dbt project. -``` -$ dbt clean --no-clean-project-files-only +```shell +dbt clean --no-clean-project-files-only ``` ## dbt clean with remote file system From 6be093be34e7324f71726859e53bb8528361da1d Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 6 Jan 2025 12:53:49 +0000 Subject: [PATCH 60/81] Update clean.md --- website/docs/reference/commands/clean.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/reference/commands/clean.md b/website/docs/reference/commands/clean.md index 24c8ffba5b4..3d42effbfcb 100644 --- a/website/docs/reference/commands/clean.md +++ b/website/docs/reference/commands/clean.md @@ -15,8 +15,8 @@ dbt clean This section will briefly explain the following flags: -- `--clean-project-files-only` (default) -- `--no-clean-project-files-only` +- [`--clean-project-files-only`](#--clean-project-files-only) (default) +- [`--no-clean-project-files-only`](#--no-clean-project-files-only) To view the list of all supported flags for the `dbt clean` command in the terminal, use the `--help` flag, which will display detailed information about the available flags you can use, including its description and usage: From d3884267ddcb85813fc657fb430abe9919badce5 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 6 Jan 2025 12:54:08 +0000 Subject: [PATCH 61/81] Update website/docs/reference/commands/clean.md --- website/docs/reference/commands/clean.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/reference/commands/clean.md b/website/docs/reference/commands/clean.md index 3d42effbfcb..24b651954ef 100644 --- a/website/docs/reference/commands/clean.md +++ b/website/docs/reference/commands/clean.md @@ -24,7 +24,7 @@ To view the list of all supported flags for the `dbt clean` command in the termi dbt clean --help ``` -### --clean-project-files-only (default) +### --clean-project-files-only By default, dbt deletes all the paths within the project directory specified in `clean-targets`. :::note From 875bf94d3275523abba1981465ef62d42f1ba0c3 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 6 Jan 2025 13:15:54 +0000 Subject: [PATCH 62/81] Update connection-profiles.md adding additional example on how to configure multiple targets in profiles.yml Resolves #867 --- .../connect-data-platform/connection-profiles.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/website/docs/docs/core/connect-data-platform/connection-profiles.md b/website/docs/docs/core/connect-data-platform/connection-profiles.md index 7feec032c21..3b791969816 100644 --- a/website/docs/docs/core/connect-data-platform/connection-profiles.md +++ b/website/docs/docs/core/connect-data-platform/connection-profiles.md @@ -39,16 +39,27 @@ jaffle_shop: dbname: jaffle_shop schema: dbt_alice threads: 4 + + prod: # additional prod target + type: postgres + host: prod.db.example.com + user: alice + password: + port: 5432 + dbname: jaffle_shop + schema: analytics + threads: 8 ``` +To add an additional target (like `prod`) to your existing `profiles.yml`, you can add another entry under the `outputs` key. ## About the `profiles.yml` file -In your `profiles.yml` file, you can store as many profiles as you need. Typically, you would have one profile for each warehouse you use. Most organizations only have one profile. +In your `profiles.yml` file, you can store as many profiles as you need. Typically, you would have one profile for each warehouse you use. Most organizations only have one profile. -For information about configuring advanced options, see [the `profiles.yml` reference page](/docs/core/connect-data-platform/profiles.yml). +or information about configuring advanced options, see [the `profiles.yml` reference page](/docs/core/connect-data-platform/profiles.yml). ## About profiles From 950e341d138763d5f60848efe7dc25da9c65c78b Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 6 Jan 2025 13:17:43 +0000 Subject: [PATCH 63/81] Update profiles.yml.md --- website/docs/docs/core/connect-data-platform/profiles.yml.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/website/docs/docs/core/connect-data-platform/profiles.yml.md b/website/docs/docs/core/connect-data-platform/profiles.yml.md index dacdc23fcd8..793352f5416 100644 --- a/website/docs/docs/core/connect-data-platform/profiles.yml.md +++ b/website/docs/docs/core/connect-data-platform/profiles.yml.md @@ -13,6 +13,8 @@ If you're using dbt Cloud, you can [connect to your data platform](/docs/cloud/c This section identifies the parts of your `profiles.yml` that aren't specific to a particular data platform. For specific connection details, refer to the relevant page for your data platform. +To add an additional target (like `prod`) to your existing `profiles.yml`, you can add another entry under the `outputs` key. + :::warning Global configs From ac043350c77b0ffeff765a4bf1e599c28eb5a8f5 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 6 Jan 2025 13:19:58 +0000 Subject: [PATCH 64/81] Update about-access.md --- website/docs/docs/cloud/manage-access/about-access.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/manage-access/about-access.md b/website/docs/docs/cloud/manage-access/about-access.md index 537494f8c02..3e1a03be30e 100644 --- a/website/docs/docs/cloud/manage-access/about-access.md +++ b/website/docs/docs/cloud/manage-access/about-access.md @@ -33,7 +33,9 @@ You can edit an existing user's license by navigating to the **Users** section o ## Groups -Groups in dbt Cloud serve much of the same purpose as they do in traditional directory tools — to gather individual users together to make bulk assignment of permissions easier. The permissions available depends on whether you're on an [Enterprise](/docs/cloud/manage-access/enterprise-permissions) or [self-service Team](/docs/cloud/manage-access/self-service-permissions) plan. +Groups in dbt Cloud serve much of the same purpose as they do in traditional directory tools — to gather individual users together to make bulk assignment of permissions easier. + +The permissions available depends on whether you're on an [Enterprise](/docs/cloud/manage-access/enterprise-permissions) or [self-service Team](/docs/cloud/manage-access/self-service-permissions) plan. - Admins use groups in dbt Cloud to assign [licenses](#licenses) and [permissions](#permissions). - The permissions are more granular than licenses, and you only assign them at the group level; _you can’t assign permissions at the user level._ From 70420952f7751adc55699e3f6988658c21c7d805 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 6 Jan 2025 14:31:51 +0000 Subject: [PATCH 65/81] Update website/docs/docs/cloud/manage-access/about-access.md Co-authored-by: nataliefiann <120089939+nataliefiann@users.noreply.github.com> --- website/docs/docs/cloud/manage-access/about-access.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/manage-access/about-access.md b/website/docs/docs/cloud/manage-access/about-access.md index 3e1a03be30e..da23183ba19 100644 --- a/website/docs/docs/cloud/manage-access/about-access.md +++ b/website/docs/docs/cloud/manage-access/about-access.md @@ -33,7 +33,7 @@ You can edit an existing user's license by navigating to the **Users** section o ## Groups -Groups in dbt Cloud serve much of the same purpose as they do in traditional directory tools — to gather individual users together to make bulk assignment of permissions easier. +Groups in dbt Cloud serve much of the same purpose as they do in traditional directory tools — to gather individual users together to make bulk assignments of permissions easier. The permissions available depends on whether you're on an [Enterprise](/docs/cloud/manage-access/enterprise-permissions) or [self-service Team](/docs/cloud/manage-access/self-service-permissions) plan. From a6eb559ff142df00024bb5a150d46750810639fc Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 6 Jan 2025 14:33:03 +0000 Subject: [PATCH 66/81] Update about-access.md --- website/docs/docs/cloud/manage-access/about-access.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/website/docs/docs/cloud/manage-access/about-access.md b/website/docs/docs/cloud/manage-access/about-access.md index da23183ba19..2cd17430b08 100644 --- a/website/docs/docs/cloud/manage-access/about-access.md +++ b/website/docs/docs/cloud/manage-access/about-access.md @@ -9,7 +9,9 @@ pagination_prev: null :::info "User access" is not "Model access" This page covers user groups and access, including: -- User licenses, permissions, group memberships, role-based access controls for projects and environments, single sign-on, and secure authentication +- User licenses, permissions, and group memberships +- Role-based access controls for projects and environments +- Single sign-on, and secure authentication For model-specific access and their availability across projects, refer to [Model access](/docs/collaborate/govern/model-access). From 79797b9f8971323e10f0d888b5eae110c46a1fb4 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Mon, 6 Jan 2025 16:49:06 +0000 Subject: [PATCH 67/81] add snippet --- website/docs/docs/dbt-cloud-environments.md | 8 +++-- .../docs/docs/deploy/deploy-environments.md | 6 ++++ website/docs/docs/deploy/deploy-jobs.md | 6 ++++ .../Environments/delete-environment-job.md | 36 ++++--------------- website/snippets/_delete-environment.md | 15 ++++++++ website/snippets/_delete-job.md | 15 ++++++++ 6 files changed, 54 insertions(+), 32 deletions(-) create mode 100644 website/snippets/_delete-environment.md create mode 100644 website/snippets/_delete-job.md diff --git a/website/docs/docs/dbt-cloud-environments.md b/website/docs/docs/dbt-cloud-environments.md index 3aa54b4aaed..b868b7a67bc 100644 --- a/website/docs/docs/dbt-cloud-environments.md +++ b/website/docs/docs/dbt-cloud-environments.md @@ -25,7 +25,6 @@ import CloudEnvInfo from '/snippets/_cloud-environments-info.md'; - ## Create a development environment To create a new dbt Cloud development environment: @@ -48,5 +47,10 @@ Deployment environments in dbt Cloud are necessary to execute scheduled jobs and Even though you can have many environments, only one of them can be the "main" deployment environment. This would be considered your "production" environment and represents your project's "source of truth", meaning it's where your most reliable and final data transformations live. - To learn more about dbt Cloud deployment environments and how to configure them, refer to the [Deployment environments](/docs/deploy/deploy-environments) page. For our best practices guide, read [dbt Cloud environment best practices](/guides/set-up-ci) for more info. + +## Delete an environment + +import DeleteEnvironment from '/snippets/_delete-environment.md'; + + diff --git a/website/docs/docs/deploy/deploy-environments.md b/website/docs/docs/deploy/deploy-environments.md index e8c7816979a..54a35d219dd 100644 --- a/website/docs/docs/deploy/deploy-environments.md +++ b/website/docs/docs/deploy/deploy-environments.md @@ -250,6 +250,12 @@ Use [extended attributes](/docs/dbt-cloud-environments#extended-attributes) to o +## Delete an environment + +import DeleteEnvironment from '/snippets/_delete-environment.md'; + + + ## Related docs - [dbt Cloud environment best practices](/guides/set-up-ci) diff --git a/website/docs/docs/deploy/deploy-jobs.md b/website/docs/docs/deploy/deploy-jobs.md index 9a0cc3cfcfa..0505f1f228d 100644 --- a/website/docs/docs/deploy/deploy-jobs.md +++ b/website/docs/docs/deploy/deploy-jobs.md @@ -128,6 +128,12 @@ To _chain_ deploy jobs together: If another job triggers your job to run, you can find a link to the upstream job in the [run details section](/docs/deploy/run-visibility#job-run-details). +## Delete a job + +import DeleteJob from '/snippets/_delete-job.md'; + + + ## Related docs - [Artifacts](/docs/deploy/artifacts) diff --git a/website/docs/faqs/Environments/delete-environment-job.md b/website/docs/faqs/Environments/delete-environment-job.md index 5b167b6df13..bcf1cf747cb 100644 --- a/website/docs/faqs/Environments/delete-environment-job.md +++ b/website/docs/faqs/Environments/delete-environment-job.md @@ -8,38 +8,14 @@ id: delete-environment-job To delete an environment or job in dbt Cloud, you must have a `developer` [license](/docs/cloud/manage-access/seats-and-users) and have the necessary [access permissions](/docs/cloud/manage-access/about-user-access). -## Delete a job +### Delete a job -To delete a job or multiple jobs in dbt Cloud: +import DeleteJob from '/snippets/_delete-job.md'; -1. Click **Deploy** on the navigation header. -2. Click **Jobs** and select the job(s) you want to delete. -3. Click **Settings** on the top right of the page and then click **Edit**. -4. Scroll to the bottom of the page and click **Delete** to delete the job.
+ -
- -
Delete a job
-
+### Delete an environment -5. Confirm your action in the **Confirm Delete** pop-up by clicking **Confirm Delete** in the bottom right to delete the job immediately. This action cannot be undone. However, you can create a new job with the same information if the deletion was made in error. +import DeleteEnvironment from '/snippets/_delete-environment.md'; -Refresh the page, and the deleted job should now be gone. If you want to delete multiple jobs, you'll need to perform these steps for each job. - -## Delete an environment - -Deleting an environment automatically deletes its associated job(s). If you want to keep those jobs, move them to a different environment first. To delete an environment in dbt Cloud: - -1. Click **Deploy** on the navigation header and then click **Environments** -2. Select the Environment you want to delete. -3. Click **Settings** on the top right of the page and then click **Edit**. -4. Scroll to the bottom of the page and click **Delete** to delete the environment.
- - - -5. Confirm your action in the **Confirm Delete** pop-up by clicking **Confirm Delete** in the bottom right to delete the environment immediately. This action cannot be undone. However, you can create a new environment with the same information if the deletion was made in error.

- - -Refresh your page, and the deleted environment should now be gone. If you want to delete multiple environments, you'll need to perform these steps to delete each one. - -If you're having any issues, feel free to [contact us](mailto:support@getdbt.com) for additional help. + diff --git a/website/snippets/_delete-environment.md b/website/snippets/_delete-environment.md new file mode 100644 index 00000000000..60339017cd2 --- /dev/null +++ b/website/snippets/_delete-environment.md @@ -0,0 +1,15 @@ +Deleting an environment automatically deletes its associated job(s). If you want to keep those jobs, move them to a different environment first. + +Follow these steps to delete an environment in dbt Cloud: + +1. Click **Deploy** on the navigation header and then click **Environments** +2. Select the environment you want to delete. +3. Click **Settings** on the top right of the page and then click **Edit**. +4. Scroll to the bottom of the page and click **Delete** to delete the environment. + + + +5. Confirm your action in the pop-up by clicking **Confirm delete** in the bottom right to delete the environment immediately. This action cannot be undone. However, you can create a new environment with the same information if the deletion was made in error. +6. Refresh your page and the deleted environment should now be gone. To delete multiple environments, you'll need to perform these steps to delete each one. + +If you're having any issues, feel free to [contact us](mailto:support@getdbt.com) for additional help. diff --git a/website/snippets/_delete-job.md b/website/snippets/_delete-job.md new file mode 100644 index 00000000000..823106bb931 --- /dev/null +++ b/website/snippets/_delete-job.md @@ -0,0 +1,15 @@ +To delete a job or multiple jobs in dbt Cloud: + +1. Click **Deploy** on the navigation header. +2. Click **Jobs** and select the job you want to delete. +3. Click **Settings** on the top right of the page and then click **Edit**. +4. Scroll to the bottom of the page and click **Delete** to delete the job.
+ +
+ +
+ +5. Confirm your action in the pop-up by clicking **Confirm delete** in the bottom right to delete the job immediately. This action cannot be undone. However, you can create a new job with the same information if the deletion was made in error. +6. Refresh the page, and the deleted job should now be gone. If you want to delete multiple jobs, you'll need to perform these steps for each job. + +If you're having any issues, feel free to [contact us](mailto:support@getdbt.com) for additional help. From 473b68021a469a5b74d5db8ed1def6bbf2361232 Mon Sep 17 00:00:00 2001 From: Tim Finkel Date: Mon, 6 Jan 2025 17:56:47 -0500 Subject: [PATCH 68/81] Fix typo (#6725) ## What are you changing in this pull request and why? Saw this typo when reading the docs. Hope this is helpful! ## Checklist - [x] I have reviewed the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ ] The topic I'm writing about is for specific dbt version(s) and I have versioned it according to the [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and/or [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content) guidelines. - [ ] I have added checklist item(s) to this list for anything anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." - [ ] The content in this PR requires a dbt release note, so I added one to the [release notes page](https://docs.getdbt.com/docs/dbt-versions/dbt-cloud-release-notes). --- website/docs/docs/build/unit-tests.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/unit-tests.md b/website/docs/docs/build/unit-tests.md index a85ffa07ed2..b7123fa35d6 100644 --- a/website/docs/docs/build/unit-tests.md +++ b/website/docs/docs/build/unit-tests.md @@ -292,7 +292,7 @@ unit_tests: There is currently no way to unit test whether the dbt framework inserted/merged the records into your existing model correctly, but [we're investigating support for this in the future](https://github.com/dbt-labs/dbt-core/issues/8664). -## Unit testing a model that depend on ephemeral model(s) +## Unit testing a model that depends on ephemeral model(s) If you want to unit test a model that depends on an ephemeral model, you must use `format: sql` for that input. From eeceb966a1a3880852b5cdaa6b2f228e63bafda5 Mon Sep 17 00:00:00 2001 From: "Leona B. Campbell" <3880403+runleonarun@users.noreply.github.com> Date: Tue, 7 Jan 2025 00:26:03 -0500 Subject: [PATCH 69/81] Fixing link (#6726) ## What are you changing in this pull request and why? Fixing link to page from blog. I forgot to check links before merging https://github.com/dbt-labs/docs.getdbt.com/pull/6725 ## Checklist - [ ] I have reviewed the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ ] The topic I'm writing about is for specific dbt version(s) and I have versioned it according to the [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and/or [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content) guidelines. - [ ] I have added checklist item(s) to this list for anything anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." - [ ] The content in this PR requires a dbt release note, so I added one to the [release notes page](https://docs.getdbt.com/docs/dbt-versions/dbt-cloud-release-notes). Co-authored-by: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> --- website/blog/2024-05-07-unit-testing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/blog/2024-05-07-unit-testing.md b/website/blog/2024-05-07-unit-testing.md index 4fea86fffb4..80b601ca8a5 100644 --- a/website/blog/2024-05-07-unit-testing.md +++ b/website/blog/2024-05-07-unit-testing.md @@ -223,7 +223,7 @@ group by 1 ### Caveats and pro-tips -See the docs for [helpful information before you begin](https://docs.getdbt.com/docs/build/unit-tests#before-you-begin), including unit testing [incremental models](https://docs.getdbt.com/docs/build/unit-tests#unit-testing-incremental-models), [models that depend on ephemeral model(s)](https://docs.getdbt.com/docs/build/unit-tests#unit-testing-a-model-that-depend-on-ephemeral-models), and platform-specific considerations like `STRUCT`s in BigQuery. In many cases, the [`sql` format](https://docs.getdbt.com/reference/resource-properties/data-formats#sql) can help solve tricky edge cases that come up. +See the docs for [helpful information before you begin](https://docs.getdbt.com/docs/build/unit-tests#before-you-begin), including unit testing [incremental models](https://docs.getdbt.com/docs/build/unit-tests#unit-testing-incremental-models), [models that depend on ephemeral model(s)](https://docs.getdbt.com/docs/build/unit-tests#unit-testing-a-model-that-depends-on-ephemeral-models), and platform-specific considerations like `STRUCT`s in BigQuery. In many cases, the [`sql` format](https://docs.getdbt.com/reference/resource-properties/data-formats#sql) can help solve tricky edge cases that come up. Another advanced topic is overcoming issues when non-deterministic factors are involved, such as a current timestamp. To ensure that the output remains consistent regardless of when the test is run, you can set a fixed, predetermined value by using the [`overrides`](https://docs.getdbt.com/reference/resource-properties/unit-test-overrides) configuration. From 9bef2a97961cfdfa769408fa1bbff8cffa53b1fe Mon Sep 17 00:00:00 2001 From: Natalie Fiann Date: Tue, 7 Jan 2025 11:30:18 +0000 Subject: [PATCH 70/81] Added multiple project call out --- website/docs/docs/build/projects.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/website/docs/docs/build/projects.md b/website/docs/docs/build/projects.md index 4732dbe6da7..dc5115fdb4d 100644 --- a/website/docs/docs/build/projects.md +++ b/website/docs/docs/build/projects.md @@ -77,6 +77,12 @@ To use the Project subdirectory option in dbt Cloud, follow these steps: After configuring the Project subdirectory option, dbt Cloud will use it as the root directory for your dbt project. This means that dbt commands, such as `dbt run` or `dbt test`, will operate on files within the specified subdirectory. If there is no `dbt_project.yml` file in the Project subdirectory, you will be prompted to initialize the dbt project. +:::info Multiple projects in dbt Cloud + +Some [plans](https://www.getdbt.com/pricing) only allow for one project, but you can have multiple projects (and cross-project references with [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro)) in Enterprise. + +::: + ## New projects You can create new projects and [share them](/docs/collaborate/git-version-control) with other people by making them available on a hosted git repository like GitHub, GitLab, and BitBucket. From e322a5cb0917e17b62bc600188a547a98b3deb71 Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 7 Jan 2025 11:52:16 +0000 Subject: [PATCH 71/81] Update website/docs/docs/build/projects.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/docs/build/projects.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/projects.md b/website/docs/docs/build/projects.md index dc5115fdb4d..c2cfb7c2d1e 100644 --- a/website/docs/docs/build/projects.md +++ b/website/docs/docs/build/projects.md @@ -79,7 +79,7 @@ After configuring the Project subdirectory option, dbt Cloud will use it as the :::info Multiple projects in dbt Cloud -Some [plans](https://www.getdbt.com/pricing) only allow for one project, but you can have multiple projects (and cross-project references with [dbt Mesh](https://docs.getdbt.com/best-practices/how-we-mesh/mesh-1-intro)) in Enterprise. +Some [plans](https://www.getdbt.com/pricing) support only one dbt project, while [Enterprise plans](https://www.getdbt.com/contact) allow multiple projects and [cross-project references](/best-practices/how-we-mesh/mesh-1-intro) with dbt Mesh. ::: From a537269dcd966f31b2293ecfa56b9040486d2cbc Mon Sep 17 00:00:00 2001 From: nataliefiann <120089939+nataliefiann@users.noreply.github.com> Date: Tue, 7 Jan 2025 11:59:54 +0000 Subject: [PATCH 72/81] Update website/docs/docs/build/projects.md Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/docs/build/projects.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/projects.md b/website/docs/docs/build/projects.md index c2cfb7c2d1e..0051b79a6b9 100644 --- a/website/docs/docs/build/projects.md +++ b/website/docs/docs/build/projects.md @@ -77,7 +77,7 @@ To use the Project subdirectory option in dbt Cloud, follow these steps: After configuring the Project subdirectory option, dbt Cloud will use it as the root directory for your dbt project. This means that dbt commands, such as `dbt run` or `dbt test`, will operate on files within the specified subdirectory. If there is no `dbt_project.yml` file in the Project subdirectory, you will be prompted to initialize the dbt project. -:::info Multiple projects in dbt Cloud +:::info Project support in dbt Cloud plans Some [plans](https://www.getdbt.com/pricing) support only one dbt project, while [Enterprise plans](https://www.getdbt.com/contact) allow multiple projects and [cross-project references](/best-practices/how-we-mesh/mesh-1-intro) with dbt Mesh. From 3760677625e4a069f798f518f1c055576d7d4014 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Tue, 7 Jan 2025 16:15:43 +0000 Subject: [PATCH 73/81] consolidate steps --- website/docs/guides/sl-snowflake-qs.md | 89 +++++++++++++------------- website/snippets/_new-sl-setup.md | 2 +- 2 files changed, 47 insertions(+), 44 deletions(-) diff --git a/website/docs/guides/sl-snowflake-qs.md b/website/docs/guides/sl-snowflake-qs.md index d70d074485f..6eada323d38 100644 --- a/website/docs/guides/sl-snowflake-qs.md +++ b/website/docs/guides/sl-snowflake-qs.md @@ -26,14 +26,33 @@ import SlSetUp from '/snippets/_new-sl-setup.md'; The [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl), powered by [MetricFlow](/docs/build/about-metricflow), simplifies the setup of key business metrics. It centralizes definitions, avoids duplicate code, and ensures easy access to metrics in downstream tools. MetricFlow helps manage company metrics easier, allowing you to define metrics in your dbt project and query them in dbt Cloud with [MetricFlow commands](/docs/build/metricflow-commands). - import SLCourses from '/snippets/_sl-course.md'; -This quickstart guide is designed for dbt Cloud users using Snowflake as their data platform. It focuses on building and defining metrics, setting up the dbt Semantic Layer in a dbt Cloud project, and querying metrics in Google Sheets. +This quickstart guide is designed for dbt Cloud users using Snowflake as their data platform. It focuses on building and defining metrics, setting up the dbt Semantic Layer in a dbt Cloud project, and querying metrics in Google Sheets. + +If you're on different data platforms, you can also follow this guide and will need to modify the setup for the specific platform. See the [Users on different platforms](#for-users-on-different-data-platforms) section for more information. + +### Prerequisites -**For users on different data platforms** +- You need a [dbt Cloud](https://www.getdbt.com/signup/) Trial, Team, or Enterprise account for all deployments. Contact your representative for Single-tenant setup; otherwise, create an account using this guide. +- Have the correct [dbt Cloud license](/docs/cloud/manage-access/seats-and-users) and [permissions](/docs/cloud/manage-access/enterprise-permissions) based on your plan: + + + - Enterprise — Developer license with Account Admin permissions. Or "Owner" with a Developer license, assigned Project Creator, Database Admin, or Admin permissions. + - Team — "Owner" access with a Developer license. + - Trial — Automatic "Owner" access under a Team plan trial. + + + +- Create a [trial Snowflake account](https://signup.snowflake.com/): + - Select the Enterprise Snowflake edition with ACCOUNTADMIN access. Consider organizational questions when choosing a cloud provider, refer to Snowflake's [Introduction to Cloud Platforms](https://docs.snowflake.com/en/user-guide/intro-cloud-platforms). + - Select a cloud provider and region. All cloud providers and regions will work so choose whichever you prefer. +- Basic understanding of SQL and dbt. For example, you've used dbt before or have completed the [dbt Fundamentals](https://learn.getdbt.com/courses/dbt-fundamentals) course. + + +### For users on different data platforms If you're using a data platform other than Snowflake, this guide is also applicable to you. You can adapt the setup for your specific platform by following the account setup and data loading instructions detailed in the following tabs for each respective platform. @@ -94,41 +113,6 @@ Open a new tab and follow these quick steps for account setup and data loading i -## Prerequisites - -- You need a [dbt Cloud](https://www.getdbt.com/signup/) Trial, Team, or Enterprise account for all deployments. Contact your representative for Single-tenant setup; otherwise, create an account using this guide. -- Have the correct [dbt Cloud license](/docs/cloud/manage-access/seats-and-users) and [permissions](/docs/cloud/manage-access/enterprise-permissions) based on your plan: - - - - Enterprise — Developer license with Account Admin permissions. Or "Owner" with a Developer license, assigned Project Creator, Database Admin, or Admin permissions. - - Team — "Owner" access with a Developer license. - - Trial — Automatic "Owner" access under a Team plan trial. - - - -- Create a [trial Snowflake account](https://signup.snowflake.com/): - - Select the Enterprise Snowflake edition with ACCOUNTADMIN access. Consider organizational questions when choosing a cloud provider, refer to Snowflake's [Introduction to Cloud Platforms](https://docs.snowflake.com/en/user-guide/intro-cloud-platforms). - - Select a cloud provider and region. All cloud providers and regions will work so choose whichever you prefer. -- Basic understanding of SQL and dbt. For example, you've used dbt before or have completed the [dbt Fundamentals](https://learn.getdbt.com/courses/dbt-fundamentals) course. - -### What you'll learn - -This guide will cover the following topics: - -- [Create a new Snowflake worksheet and set up your environment](/guides/sl-snowflake-qs?step=3) -- [Load sample data into your Snowflake account](/guides/sl-snowflake-qs?step=4) -- [Connect dbt Cloud to Snowflake](/guides/sl-snowflake-qs?step=5) -- [Set up a dbt Cloud managed repository](/guides/sl-snowflake-qs?step=6) -- [Initialize a dbt Cloud project and start developing](/guides/sl-snowflake-qs?step=7) -- [Build your dbt Cloud project](/guides/sl-snowflake-qs?step=8) -- [Create a semantic model in dbt Cloud](/guides/sl-snowflake-qs?step=9) -- [Define metrics in dbt Cloud](/guides/sl-snowflake-qs?step=10) -- [Add second semantic model](/guides/sl-snowflake-qs?step=11) -- [Test and query metrics in dbt Cloud](/guides/sl-snowflake-qs?step=12) -- [Run a production job in dbt Cloud](/guides/sl-snowflake-qs?step=13) -- [Set up dbt Semantic Layer in dbt Cloud](/guides/sl-snowflake-qs?step=14) -- [Connect and query metrics with Google Sheets](/guides/sl-snowflake-qs?step=15) - ## Create new Snowflake worksheet and set up environment 1. Log in to your [trial Snowflake account](https://signup.snowflake.com). @@ -158,7 +142,7 @@ create schema raw.jaffle_shop; create schema raw.stripe; ``` -## Load data into Snowflake +### Load data into Snowflake Now that your environment is set up, you can start loading data into it. You will be working within the raw database, using the `jaffle_shop` and stripe schemas to organize your tables. 1. Create customer table. First, delete all contents (empty) in the Editor of the Snowflake worksheet. Then, run this SQL command to create the customer table in the `jaffle_shop` schema: @@ -332,12 +316,16 @@ Using Partner Connect allows you to create a complete dbt account with your [Sno -## Set up a dbt Cloud managed repository +## Set up dbt Cloud project + +In this section, you will set up a dbt Cloud managed repository and initialize your dbt project to start developing. + +### Set up a dbt Cloud managed repository If you used Partner Connect, you can skip to [initializing your dbt project](#initialize-your-dbt-project-and-start-developing) as Partner Connect provides you with a [managed repository](/docs/cloud/git/managed-repository). Otherwise, you will need to create your repository connection. -## Initialize your dbt project and start developing +### Initialize your dbt project This guide assumes you use the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) to develop your dbt project, define metrics, and query and preview metrics using [MetricFlow commands](/docs/build/metricflow-commands). Now that you have a repository configured, you can initialize your project and start development in dbt Cloud using the IDE: @@ -637,12 +625,15 @@ semantic_models: +### Semantic model components + The following sections explain [dimensions](/docs/build/dimensions), [entities](/docs/build/entities), and [measures](/docs/build/measures) in more detail, showing how they each play a role in semantic models. - [Entities](#entities) act as unique identifiers (like ID columns) that link data together from different tables. - [Dimensions](#dimensions) categorize and filter data, making it easier to organize. - [Measures](#measures) calculates data, providing valuable insights through aggregation. + ### Entities [Entities](/docs/build/semantic-models#entities) are a real-world concept in a business, serving as the backbone of your semantic model. These are going to be ID columns (like `order_id`) in our semantic models. These will serve as join keys to other semantic models. @@ -753,7 +744,11 @@ semantic_models: -## Define metrics +## Define metrics and add a second semantic model + +In this section, you will [define metrics](#define-metrics) and [add a second semantic model](#add-second-semantic-model-to-your-project) to your project. + +### Define metrics [Metrics](/docs/build/metrics-overview) are the language your business users speak and measure business performance. They are an aggregation over a column in your warehouse that you enrich with dimensional cuts. @@ -870,7 +865,7 @@ metrics: -## Add second semantic model to your project +### Add second semantic model to your project Great job, you've successfully built your first semantic model! It has all the required elements: entities, dimensions, measures, and metrics. @@ -955,6 +950,14 @@ https://github.com/dbt-labs/docs.getdbt.com/blob/current/website/snippets/_sl-ru ## Set up dbt Semantic Layer +In this section, you will learn how to set up the dbt Semantic Layer, add credentials, and create service tokens. This section goes over the following topics: + +- [Select environment](#1-select-environment) +- [Add a credential and create service tokens](#2-add-a-credential-and-create-service-tokens) +- [View connection detail](#3-view-connection-detail) +- [Add more credentials](#4-add-more-credentials) +- [Delete configuration](#delete-configuration) + --- 🚀 Deployment available! Here are the direct links to the updated files: - https://docs-getdbt-com-git-trouze-check-cols-all-dbt-labs.vercel.app/reference/resource-configs/check_cols --- website/docs/reference/resource-configs/check_cols.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/website/docs/reference/resource-configs/check_cols.md b/website/docs/reference/resource-configs/check_cols.md index f7c6b85d372..97f2fdfcf96 100644 --- a/website/docs/reference/resource-configs/check_cols.md +++ b/website/docs/reference/resource-configs/check_cols.md @@ -122,8 +122,7 @@ snapshots: schema: snapshots unique_key: id strategy: check - check_cols: - - all + check_cols: all ``` From d03d2ece29a92466d915fa4b3ffdb6ebd7a5ec8b Mon Sep 17 00:00:00 2001 From: "Leona B. Campbell" <3880403+runleonarun@users.noreply.github.com> Date: Tue, 7 Jan 2025 15:54:59 -0800 Subject: [PATCH 77/81] Update usage-stats.md --- website/docs/reference/global-configs/usage-stats.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/docs/reference/global-configs/usage-stats.md b/website/docs/reference/global-configs/usage-stats.md index 73610c29586..aa9dacd6658 100644 --- a/website/docs/reference/global-configs/usage-stats.md +++ b/website/docs/reference/global-configs/usage-stats.md @@ -18,11 +18,11 @@ For full transparency, you can see all the event definitions in [`tracking.py`]( - dbt Cloud has telemetry enabled by default to help us enhance the user experience and improve the product by using real user feedback and usage patterns. While it cannot be disabled, we ensure the data is [secure](https://www.getdbt.com/security) and used responsibly. Collecting this data enables us to provide a better product experience, including improvements to the performance of dbt. -- dbt Core users have telemetry enabled by default to help us understand usage patterns and improve the product. Users can opt out of event tracking at any time by adding the following to your `profiles.yml` file: +- dbt Core users have telemetry enabled by default to help us understand usage patterns and improve the product. You can opt out of event tracking at any time by adding the following to your `dbt_project.yml` file: ```yaml - config: - send_anonymous_usage_stats: False + flags: + send_anonymous_usage_stats: false ``` dbt Core users can also use the `DO_NOT_TRACK` environment variable to enable or disable sending anonymous data. For more information, see [Environment variables](/docs/build/environment-variables). From 03303ae51b2df9e9128d8aa0c7d484d9f002645f Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Wed, 8 Jan 2025 13:27:40 +0000 Subject: [PATCH 78/81] Update usage-stats.md add file name for explicitness --- website/docs/reference/global-configs/usage-stats.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/website/docs/reference/global-configs/usage-stats.md b/website/docs/reference/global-configs/usage-stats.md index aa9dacd6658..1c06105cde9 100644 --- a/website/docs/reference/global-configs/usage-stats.md +++ b/website/docs/reference/global-configs/usage-stats.md @@ -20,10 +20,13 @@ For full transparency, you can see all the event definitions in [`tracking.py`]( - dbt Core users have telemetry enabled by default to help us understand usage patterns and improve the product. You can opt out of event tracking at any time by adding the following to your `dbt_project.yml` file: + + ```yaml flags: send_anonymous_usage_stats: false ``` + dbt Core users can also use the `DO_NOT_TRACK` environment variable to enable or disable sending anonymous data. For more information, see [Environment variables](/docs/build/environment-variables). From 495a66d4426ae036cd347fa79959359e68e92832 Mon Sep 17 00:00:00 2001 From: Dan Horner <107440088+datapackdan@users.noreply.github.com> Date: Wed, 8 Jan 2025 14:39:46 -0500 Subject: [PATCH 79/81] Update incremental-models.md (#6739) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit update conditions for is_incremental() to evaluate to true based on Issue 5279 ## What are you changing in this pull request and why? This update is to address [Issue 5279 - Understand the is_incremental() macro](https://github.com/dbt-labs/docs.getdbt.com/issues/5279) I believe the original intent of the first two conditions was to say: "the model has to exist as a table" however saying this using two lines feels redundant and I know from training sessions that users react much the same way as the author of this issue. I'm proposing the following wording: Revised The `is_incremental()` macro powers incremental materializations. It will return `True` if _all_ of the following conditions are met: - The model must already exist _as a table_ in the database - The `full-refresh` flag _is not_ passed - The running model is configured with `materialized='incremental'` Original (for reference) The `is_incremental()` macro powers incremental materializations. It will return `True` if _all_ of the following conditions are met: - The model must already exist in the database - The destination table already exists in the database - The `full-refresh` flag _is not_ passed - The running model is configured with `materialized='incremental'` ## Checklist - [x] I have reviewed the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ ] The topic I'm writing about is for specific dbt version(s) and I have versioned it according to the [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and/or [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content) guidelines. - [ ] I have added checklist item(s) to this list for anything anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." - [ ] The content in this PR requires a dbt release note, so I added one to the [release notes page](https://docs.getdbt.com/docs/dbt-versions/dbt-cloud-release-notes). --- 🚀 Deployment available! Here are the direct links to the updated files: - https://docs-getdbt-com-git-datapackdan-patch-issue-5279-dbt-labs.vercel.app/docs/build/incremental-models --- website/docs/docs/build/incremental-models.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/website/docs/docs/build/incremental-models.md b/website/docs/docs/build/incremental-models.md index 0560797c9bc..6b3f3f7843c 100644 --- a/website/docs/docs/build/incremental-models.md +++ b/website/docs/docs/build/incremental-models.md @@ -34,8 +34,7 @@ To use incremental models, you also need to tell dbt: The `is_incremental()` macro powers incremental materializations. It will return `True` if _all_ of the following conditions are met: -- The model must already exist in the database -- The destination table already exists in the database +- The model must already exist as a table in the database - The `full-refresh` flag _is not_ passed - The running model is configured with `materialized='incremental'` From 1c26bb23bcdc140634dcbad9a42cd973a92b6e31 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Wed, 8 Jan 2025 19:49:25 +0000 Subject: [PATCH 80/81] add plus operator example to tags doc (#6737) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit this pr adds an example combining tags and the plus operator in the `tags` doc and the `+ operator` doc. Resolves #6735 --- 🚀 Deployment available! Here are the direct links to the updated files: - https://docs-getdbt-com-git-mirnawong1-patch-26-dbt-labs.vercel.app/reference/node-selection/graph-operators - https://docs-getdbt-com-git-mirnawong1-patch-26-dbt-labs.vercel.app/reference/resource-configs/tags --------- Co-authored-by: nataliefiann <120089939+nataliefiann@users.noreply.github.com> --- .../reference/node-selection/graph-operators.md | 10 +++++++--- website/docs/reference/resource-configs/tags.md | 17 ++++++++++++----- 2 files changed, 19 insertions(+), 8 deletions(-) diff --git a/website/docs/reference/node-selection/graph-operators.md b/website/docs/reference/node-selection/graph-operators.md index 455af9aa137..6ea4b9b7fa9 100644 --- a/website/docs/reference/node-selection/graph-operators.md +++ b/website/docs/reference/node-selection/graph-operators.md @@ -3,15 +3,19 @@ title: "Graph operators" --- ### The "plus" operator -If placed at the front of the model selector, `+` will select all ancestors of the selected model and the model itself. If placed at the end of the string, `+` will select all descendants of the selected model and the model itself. +The `+` operator expands your selection to include ancestors (upstream dependencies) or descendants (downstream dependencies) of a resource. This operator works for individual models, tags, and other resources. +- Placed after a model/resource — Includes the resource itself and all its descendants (downstream dependencies). +- Placed before a model/resource — Includes the resource itself and all its ancestors (upstream dependencies). +- Placed on both sides of a model/resource — Includes the resource itself, all its ancestors, and all its descendants. - ```bash +```bash dbt run --select "my_model+" # select my_model and all descendants dbt run --select "+my_model" # select my_model and all ancestors dbt run --select "+my_model+" # select my_model, and all of its ancestors and descendants - ``` +``` +You can use it with selectors for a more specific scope in your commands. You can also combine it with [`--exclude`](/reference/node-selection/exclude) flag for even more finer control over what gets included in your command. ### The "n-plus" operator diff --git a/website/docs/reference/resource-configs/tags.md b/website/docs/reference/resource-configs/tags.md index c222df8c1ae..505a33a00f7 100644 --- a/website/docs/reference/resource-configs/tags.md +++ b/website/docs/reference/resource-configs/tags.md @@ -113,11 +113,18 @@ resource_type: Apply a tag (or list of tags) to a resource. These tags can be used as part of the [resource selection syntax](/reference/node-selection/syntax), when running the following commands: -- `dbt run --select tag:my_tag` -- `dbt build --select tag:my_tag` -- `dbt seed --select tag:my_tag` -- `dbt snapshot --select tag:my_tag` -- `dbt test --select tag:my_tag` (indirectly runs all tests associated with the models that are tagged) +- `dbt run --select tag:my_tag` — Run all models tagged with a specific tag. +- `dbt build --select tag:my_tag` — Build all resources tagged with a specific tag. +- `dbt seed --select tag:my_tag` — Seed all resources tagged with a specific tag. +- `dbt snapshot --select tag:my_tag` — Snapshot all resources tagged with a specific tag. +- `dbt test --select tag:my_tag` — Indirectly runs all tests associated with the models that are tagged. + +#### Using tags with the `+` operator +You can use the [`+` operator](/reference/node-selection/graph-operators#the-plus-operator) to include upstream or downstream dependencies in your `tag` selection: +- `dbt run --select tag:my_tag+` — Run models tagged with `my_tag` and all their downstream dependencies. +- `dbt run --select +tag:my_tag` — Run models tagged with `my_tag` and all their upstream dependencies. +- `dbt run --select +model_name+` — Run a model, its upstream dependencies, and its downstream dependencies. +- `dbt run --select tag:my_tag+ --exclude tag:exclude_tag` — Run model tagged with `my_tag` and their downstream dependencies, and exclude models tagged with `exclude_tag`, regardless of their dependencies. ## Examples ### Use tags to run parts of your project From 13c396840e4f32cc69cad90b8c09f394c6ca9643 Mon Sep 17 00:00:00 2001 From: Naka Masato Date: Thu, 9 Jan 2025 05:30:42 +0900 Subject: [PATCH 81/81] chore: remove duplicated 'the' (#6740) ## What are you changing in this pull request and why? `the the` -> `the` ## Checklist - [x] I have reviewed the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ ] The topic I'm writing about is for specific dbt version(s) and I have versioned it according to the [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and/or [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content) guidelines. - [ ] I have added checklist item(s) to this list for anything anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." - [ ] The content in this PR requires a dbt release note, so I added one to the [release notes page](https://docs.getdbt.com/docs/dbt-versions/dbt-cloud-release-notes). Co-authored-by: Leona B. Campbell <3880403+runleonarun@users.noreply.github.com> --- ...2022-04-19-dbt-cloud-postman-collection.md | 12 ++--- ...-05-17-stakeholder-friendly-model-names.md | 14 +++--- .../cloud/manage-access/set-up-sso-okta.md | 4 +- .../manage-access/set-up-sso-saml-2.0.md | 2 +- .../docs/docs/collaborate/explore-projects.md | 48 +++++++++---------- .../99-dbt-cloud-changelog-2019-2020.md | 6 +-- .../dbt-jinja-functions/run_started_at.md | 8 ++-- .../resource-configs/bigquery-configs.md | 10 ++-- 8 files changed, 52 insertions(+), 52 deletions(-) diff --git a/website/blog/2022-04-19-dbt-cloud-postman-collection.md b/website/blog/2022-04-19-dbt-cloud-postman-collection.md index 7ea81e89181..16ecb04670d 100644 --- a/website/blog/2022-04-19-dbt-cloud-postman-collection.md +++ b/website/blog/2022-04-19-dbt-cloud-postman-collection.md @@ -19,7 +19,7 @@ is_featured: true The dbt Cloud API has well-documented endpoints for creating, triggering and managing dbt Cloud jobs. But there are other endpoints that aren’t well documented yet, and they’re extremely useful for end-users. These endpoints exposed by the API enable organizations not only to orchestrate jobs, but to manage their dbt Cloud accounts programmatically. This creates some really interesting capabilities for organizations to scale their dbt Cloud implementations. -The main goal of this article is to spread awareness of these endpoints as the docs are being built & show you how to use them. +The main goal of this article is to spread awareness of these endpoints as the docs are being built & show you how to use them. @@ -45,7 +45,7 @@ Beyond the day-to-day process of managing their dbt Cloud accounts, many organiz *Below this you’ll find a series of example requests - use these to guide you or [check out the Postman Collection](https://dbtlabs.postman.co/workspace/Team-Workspace~520c7ac4-3895-4779-8bc3-9a11b5287c1c/request/12491709-23cd2368-aa58-4c9a-8f2d-e8d56abb6b1dlinklink) to try it out yourself.* -## Appendix +## Appendix ### Examples of how to use the Postman Collection @@ -55,7 +55,7 @@ Let’s run through some examples on how to make good use of this Postman Collec One common question we hear from customers is “How can we migrate resources from one dbt Cloud project to another?” Often, they’ll create a development project, in which users have access to the UI and can manually make changes, and then migrate selected resources from the development project to a production project once things are ready. -There are several reasons one might want to do this, including: +There are several reasons one might want to do this, including: - Probably the most common is separating dev/test/prod environments across dbt Cloud projects to enable teams to build manually in a development project, and then automatically migrate those environments & jobs to a production project. - Building “starter projects” they can deploy as templates for new teams onboarding to dbt from a learning standpoint. @@ -90,10 +90,10 @@ https://cloud.getdbt.com/api/v3/accounts/28885/projects/86704/environments/75286 #### Push the environment to the production project -We take the response from the GET request above, and then to the following: +We take the response from the GET request above, and then to the following: 1. Adjust some of the variables for the new environment: - - Change the the value of the “project_id” field from 86704 to 86711 + - Change the value of the “project_id” field from 86704 to 86711 - Change the value of the “name” field from “dev-staging” to “production–api-generated” - Set the “custom_branch” field to “main” @@ -116,7 +116,7 @@ We take the response from the GET request above, and then to the following: } ``` -3. Note the environment ID returned in the response, as we’ll use to create a dbt Cloud job in the next step +3. Note the environment ID returned in the response, as we’ll use to create a dbt Cloud job in the next step #### Pull the job definition from the dev project diff --git a/website/blog/2022-05-17-stakeholder-friendly-model-names.md b/website/blog/2022-05-17-stakeholder-friendly-model-names.md index 39107035465..7170770106a 100644 --- a/website/blog/2022-05-17-stakeholder-friendly-model-names.md +++ b/website/blog/2022-05-17-stakeholder-friendly-model-names.md @@ -29,7 +29,7 @@ In this article, we’ll take a deeper look at why model naming conventions are >“[Data folks], what we [create in the database]… echoes in eternity.” -Max(imus, Gladiator) -Analytics Engineers are often centrally located in the company, sandwiched between data analysts and data engineers. This means everything AEs create might be read and need to be understood by both an analytics or customer-facing team and by teams who spend most of their time in code and the database. Depending on the audience, the scope of access differs, which means the user experience and context changes. Let’s elaborate on what that experience might look like by breaking end-users into two buckets: +Analytics Engineers are often centrally located in the company, sandwiched between data analysts and data engineers. This means everything AEs create might be read and need to be understood by both an analytics or customer-facing team and by teams who spend most of their time in code and the database. Depending on the audience, the scope of access differs, which means the user experience and context changes. Let’s elaborate on what that experience might look like by breaking end-users into two buckets: - Analysts / BI users - Analytics engineers / Data engineers @@ -49,13 +49,13 @@ Here we have drag and drop functionality and a skin over top of the underlying ` **How model names can make this painful:** The end users might not even know what tables the data refers to, as potentially everything is joined by the system and they don’t need to write their own queries. If model names are chosen poorly, there is a good chance that the BI layer on top of the database tables has been renamed to something more useful for the analysts. This adds an extra step of mental complexity in tracing the lineage from data model to BI. -#### Read only access to the dbt Cloud IDE docs +#### Read only access to the dbt Cloud IDE docs If Analysts want more context via documentation, they may traverse back to the dbt layer and check out the data models in either the context of the Project or Database. In the Project view, they will see the data models in the folder hierarchy present in your project’s repository. In the Database view you will see the output of the data models as present in your database, ie. `database / schema / object`. ![A screenshot depicting the dbt Cloud IDE menu's Database view which shows you the output of your data models. Next to this view, is the Project view.](/img/blog/2022-05-17-stakeholder-friendly-model-names/project-view.png) **How model names can make this painful:** -For the Project view, generally abstracted department or organizational structures as folder names presupposes the reader/engineer knows what is contained within the folder beforehand or what that department actually does, or promotes haphazard clicking to open folders to see what is within. Organizing the final outputs by business unit or analytics function is great for end users but doesn't accurately represent all the sources and references that had to come together to build this output, as they often live in another folder. +For the Project view, generally abstracted department or organizational structures as folder names presupposes the reader/engineer knows what is contained within the folder beforehand or what that department actually does, or promotes haphazard clicking to open folders to see what is within. Organizing the final outputs by business unit or analytics function is great for end users but doesn't accurately represent all the sources and references that had to come together to build this output, as they often live in another folder. For the Database view, pray your team has been declaring a logical schema bucketing, or a logical model naming convention, otherwise you will have a long, alphabetized list of database objects to scroll through, where staging, intermediate, and final output models are all intermixed. Clicking into a data model and viewing the documentation is helpful, but you would need to check out the DAG to see where the model lives in the overall flow. @@ -63,7 +63,7 @@ For the Database view, pray your team has been declaring a logical schema bucket If they have access to Worksheets, SQL runner, or another way to write ad hoc sql queries, then they will have access to the data models as present in your database, ie. `database / schema / object`, but with less documentation attached, and more proclivity towards querying tables to check out their contents, which costs time and money. -![A screenshot of the the SQL Runner menu within Looker showcasing the dropdown list of all data models present in the database.](/img/blog/2022-05-17-stakeholder-friendly-model-names/data-warehouse-dropdown.png) +![A screenshot of the SQL Runner menu within Looker showcasing the dropdown list of all data models present in the database.](/img/blog/2022-05-17-stakeholder-friendly-model-names/data-warehouse-dropdown.png) **How model names can make this painful:** Without proper naming conventions, you will encounter `analytics.order`, `analytics.orders`, `analytics.orders_new` and not know which one is which, so you will open up a scratch statement tab and attempt to figure out which is correct: @@ -73,9 +73,9 @@ Without proper naming conventions, you will encounter `analytics.order`, `analyt -- select * from analytics.orders limit 10 select * from analytics.orders_new limit 10 ``` -Hopefully you get it right via sampling queries, or eventually find out there is a true source of truth defined in a totally separate area: `core.dim_orders`. +Hopefully you get it right via sampling queries, or eventually find out there is a true source of truth defined in a totally separate area: `core.dim_orders`. -The problem here is the only information you can use to determine what data is within an object or the purpose of the object is within the schema and model name. +The problem here is the only information you can use to determine what data is within an object or the purpose of the object is within the schema and model name. ### The engineer’s user experience @@ -98,7 +98,7 @@ There is not much worse than spending all week developing on a task, submitting This is largely the same as the Analyst experience above, except they created the data models or are aware of their etymologies. They are likely more comfortable writing ad hoc queries, but also have the ability to make changes, which adds a layer of thought processing when working. **How model names can make this painful:** -It takes time to become a subject matter expert in the database. You will need to know which schema a subject lives in, what tables are the source of truth and/or output models, versus experiments, outdated objects, or building blocks used along the way. Working within this context, engineers know the history and company lore behind why a table was named that way or how its purpose may differ slightly from its name, but they also have the ability to make changes. +It takes time to become a subject matter expert in the database. You will need to know which schema a subject lives in, what tables are the source of truth and/or output models, versus experiments, outdated objects, or building blocks used along the way. Working within this context, engineers know the history and company lore behind why a table was named that way or how its purpose may differ slightly from its name, but they also have the ability to make changes. Change management is hard; how many places would you need to update, rename, re-document, and retest to fix a poor naming choice from long ago? It is a daunting position, which can create internal strife when constrained for time over whether we should continually revamp and refactor for maintainability or focus on building new models in the same pattern as before. diff --git a/website/docs/docs/cloud/manage-access/set-up-sso-okta.md b/website/docs/docs/cloud/manage-access/set-up-sso-okta.md index fda32f118ef..83c9f6492c6 100644 --- a/website/docs/docs/cloud/manage-access/set-up-sso-okta.md +++ b/website/docs/docs/cloud/manage-access/set-up-sso-okta.md @@ -152,7 +152,7 @@ the integration between Okta and dbt Cloud. ## Configuration in dbt Cloud -To complete setup, follow the steps below in dbt Cloud. +To complete setup, follow the steps below in dbt Cloud. ### Supplying credentials @@ -182,7 +182,7 @@ configured in the steps above. 21. Click **Save** to complete setup for the Okta integration. From here, you can navigate to the URL generated for your account's _slug_ to - test logging in with Okta. Additionally, users added the the Okta app + test logging in with Okta. Additionally, users added the Okta app will be able to log in to dbt Cloud from Okta directly. diff --git a/website/docs/docs/cloud/manage-access/set-up-sso-saml-2.0.md b/website/docs/docs/cloud/manage-access/set-up-sso-saml-2.0.md index 34c1a91fbee..ca93d81badf 100644 --- a/website/docs/docs/cloud/manage-access/set-up-sso-saml-2.0.md +++ b/website/docs/docs/cloud/manage-access/set-up-sso-saml-2.0.md @@ -312,7 +312,7 @@ Follow these steps to set up single sign-on (SSO) with dbt Cloud: 12. Click **Edit** in the Basic SAML Configuration section. - + 13. Use the following table to complete the required fields and connect to dbt: diff --git a/website/docs/docs/collaborate/explore-projects.md b/website/docs/docs/collaborate/explore-projects.md index 627b255cd78..e21bd507e51 100644 --- a/website/docs/docs/collaborate/explore-projects.md +++ b/website/docs/docs/collaborate/explore-projects.md @@ -7,7 +7,7 @@ pagination_next: "docs/collaborate/data-health-signals" pagination_prev: null --- -With dbt Explorer, you can view your project's [resources](/docs/build/projects) (such as models, tests, and metrics), their lineage, and [model consumption](/docs/collaborate/auto-exposures) to gain a better understanding of its latest production state. Navigate and manage your projects within dbt Cloud to help you and other data developers, analysts, and consumers discover and leverage your dbt resources. +With dbt Explorer, you can view your project's [resources](/docs/build/projects) (such as models, tests, and metrics), their lineage, and [model consumption](/docs/collaborate/auto-exposures) to gain a better understanding of its latest production state. Navigate and manage your projects within dbt Cloud to help you and other data developers, analysts, and consumers discover and leverage your dbt resources. import ExplorerCourse from '/snippets/_explorer-course-link.md'; @@ -41,23 +41,23 @@ dbt Explorer uses the metadata provided by the [Discovery API](/docs/dbt-cloud-a - dbt Explorer automatically retrieves the metadata updates after each job run in the production or staging deployment environment so it always has the latest results for your project. This includes deploy and merge jobs. - Note that CI jobs do not update dbt Explorer. This is because they don't reflect the production state and don't provide the necessary metadata updates. -- To view a resource and its metadata, you must define the resource in your project and run a job in the production or staging environment. -- The resulting metadata depends on the [commands](/docs/deploy/job-commands) executed by the jobs. +- To view a resource and its metadata, you must define the resource in your project and run a job in the production or staging environment. +- The resulting metadata depends on the [commands](/docs/deploy/job-commands) executed by the jobs. | To view in Explorer | You must successfully run | |---------------------|---------------------------| | Model lineage, details, or results | [dbt run](/reference/commands/run) or [dbt build](/reference/commands/build) on a given model within a job in the environment | | Columns and statistics for models, sources, and snapshots| [dbt docs generate](/reference/commands/cmd-docs) within [a job](/docs/collaborate/build-and-view-your-docs) in the environment | -| Test results | [dbt test](/reference/commands/test) or [dbt build](/reference/commands/build) within a job in the environment | +| Test results | [dbt test](/reference/commands/test) or [dbt build](/reference/commands/build) within a job in the environment | | Source freshness results | [dbt source freshness](/reference/commands/source#dbt-source-freshness) within a job in the environment | | Snapshot details | [dbt snapshot](/reference/commands/snapshot) or [dbt build](/reference/commands/build) within a job in the environment | | Seed details | [dbt seed](/reference/commands/seed) or [dbt build](/reference/commands/build) within a job in the environment | -Richer and more timely metadata will become available as dbt Cloud evolves. +Richer and more timely metadata will become available as dbt Cloud evolves. ## Explore your project's lineage graph {#project-lineage} -dbt Explorer provides a visualization of your project’s DAG that you can interact with. To access the project's full lineage graph, select **Overview** in the left sidebar and click the **Explore Lineage** button on the main (center) section of the page. +dbt Explorer provides a visualization of your project’s DAG that you can interact with. To access the project's full lineage graph, select **Overview** in the left sidebar and click the **Explore Lineage** button on the main (center) section of the page. If you don't see the project lineage graph immediately, click **Render Lineage**. It can take some time for the graph to render depending on the size of your project and your computer’s available memory. The graph of very large projects might not render so you can select a subset of nodes by using selectors, instead. @@ -78,7 +78,7 @@ To explore the lineage graphs of tests and macros, view [their resource details - Refocus on the node and it upstream nodes only - View the node's [resource details](#view-resource-details) page - Select a resource to highlight its relationship with other resources in your project. A panel opens on the graph’s right-hand side that displays a high-level summary of the resource’s details. The side panel includes a **General** tab for information like description, materialized type, and other details. In the side panel's upper right corner: - - Click the View Resource icon to [view the resource details](#view-resource-details). + - Click the View Resource icon to [view the resource details](#view-resource-details). - Click the [Open in IDE](#open-in-ide) icon to examine the resource using the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud). - Click the Copy Link to Page icon to copy the page's link to your clipboard. - Use [selectors](/reference/node-selection/methods) (in the search bar) to select specific resources or a subset of the DAG. This can help narrow the focus on the resources that interest you. All selectors are available for use, except those requiring a state comparison (result, source status, and state). You can also use the `--exclude` and the `--select` flag (which is optional). Examples: @@ -91,7 +91,7 @@ To explore the lineage graphs of tests and macros, view [their resource details - `+snowplow_sessions +fct_orders` — Use space-delineated arguments for a union operation. Returns resources that are upstream nodes of either `snowplow_sessions` or `fct_orders`. - [View resource details](#view-resource-details) by selecting a node (double-clicking) in the graph. -- Click **Lenses** (lower right corner of the graph) to use Explorer's [lenses](#lenses) feature. +- Click **Lenses** (lower right corner of the graph) to use Explorer's [lenses](#lenses) feature. @@ -120,7 +120,7 @@ A resource in your project is characterized by resource type, materialization ty - **Marts** — A model with the prefix `fct_` or `dim_` or a model that lives in the `/marts/` subdirectory. - **Intermediate** — A model with the prefix `int_`. Or, a model that lives in the `/int/` or `/intermediate/` subdirectory. - **Staging** — A model with the prefix `stg_`. Or, a model that lives in the `/staging/` subdirectory. -- **Test status**: The status from the latest execution of the tests that ran again this resource. In the case that a model has multiple tests with different results, the lens reflects the 'worst case' status. +- **Test status**: The status from the latest execution of the tests that ran again this resource. In the case that a model has multiple tests with different results, the lens reflects the 'worst case' status. - **Consumption query history**: The number of queries against this resource over a given time period. @@ -159,7 +159,7 @@ The **Filters** side panel becomes available after you perform a keyword search. - [Model materialization](/docs/build/materializations) (like view, table) - [Tags](/reference/resource-configs/tags) (supports multi-select) -Under the the **Models** option, you can filter on model properties (access or materialization type). Also available are **Advanced** options, where you can limit the search results to column name, model code, and more. +Under the **Models** option, you can filter on model properties (access or materialization type). Also available are **Advanced** options, where you can limit the search results to column name, model code, and more. @@ -170,20 +170,20 @@ Example of results from searching on the keyword `customers` and applying the fi ## Browse with the sidebar -From the sidebar, you can browse your project's resources, its file tree, and the database. +From the sidebar, you can browse your project's resources, its file tree, and the database. - **Resources** tab — All resources in the project organized by type. Select any resource type in the list and all those resources in the project will display as a table in the main section of the page. For a description on the different resource types (like models, metrics, and so on), refer to [About dbt projects](/docs/build/projects). - [Data health signals](/docs/collaborate/data-health-signals) are visible to the right of the resource name under the **Health** column. - **File Tree** tab — All resources in the project organized by the file in which they are defined. This mirrors the file tree in your dbt project repository. -- **Database** tab — All resources in the project organized by the database and schema in which they are built. This mirrors your data platform's structure that represents the [applied state](/docs/dbt-cloud-apis/project-state) of your project. +- **Database** tab — All resources in the project organized by the database and schema in which they are built. This mirrors your data platform's structure that represents the [applied state](/docs/dbt-cloud-apis/project-state) of your project. ## Open in IDE -If you have been assigned a [developer license](/docs/cloud/manage-access/about-user-access#license-based-access-control), you can open the resource in the [IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) directly from Explorer. For example, the IDE opens all the corresponding files for the model. This includes the model's SQL or Python definition and any YAML files that include an entry for that model. The feature is available from the [full lineage graph](#example-of-full-lineage-graph) and the [resource's details view](#example-of-model-details). +If you have been assigned a [developer license](/docs/cloud/manage-access/about-user-access#license-based-access-control), you can open the resource in the [IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) directly from Explorer. For example, the IDE opens all the corresponding files for the model. This includes the model's SQL or Python definition and any YAML files that include an entry for that model. The feature is available from the [full lineage graph](#example-of-full-lineage-graph) and the [resource's details view](#example-of-model-details). -Here's an example of the Open in IDE icon in the upper right corner of the resource details page. The icon is inactive (grayed out) if you haven't been assigned a developer license. +Here's an example of the Open in IDE icon in the upper right corner of the resource details page. The icon is inactive (grayed out) if you haven't been assigned a developer license. @@ -192,9 +192,9 @@ Here's an example of the Open in IDE icon in the upper right corner of the resou If models in the project are versioned, you can see which [version of the model](/docs/collaborate/govern/model-versions) is being applied — `prerelease`, `latest`, and `old` — in the title of the model’s details page and in the model list from the sidebar. ## View resource details {#view-resource-details} -You can view the definition and latest run results of any resource in your project. To find a resource and view its details, you can interact with the lineage graph, use search, or browse the catalog. +You can view the definition and latest run results of any resource in your project. To find a resource and view its details, you can interact with the lineage graph, use search, or browse the catalog. -The details (metadata) available to you depends on the resource’s type, its definition, and the [commands](/docs/deploy/job-commands) that run within jobs in the production environment. +The details (metadata) available to you depends on the resource’s type, its definition, and the [commands](/docs/deploy/job-commands) that run within jobs in the production environment. In the upper right corner of the resource details page, you can: - Click the [Open in IDE](#open-in-ide) icon to examine the resource using the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud). @@ -203,22 +203,22 @@ In the upper right corner of the resource details page, you can: - **Data health signals** — [Data health signals](/docs/collaborate/data-health-signals) offer a quick, at-a-glance view of data health. These icons indicate whether a model is Healthy, Caution, Degraded, or Unknown. Hover over an icon to view detailed information about the model's health. -- **Status bar** (below the page title) — Information on the last time the model ran, whether the run was successful, how the data is materialized, number of rows, and the size of the model. +- **Status bar** (below the page title) — Information on the last time the model ran, whether the run was successful, how the data is materialized, number of rows, and the size of the model. - **General** tab includes: - **Lineage** graph — The model’s lineage graph that you can interact with. The graph includes one upstream node and one downstream node from the model. Click the Expand icon in the graph's upper right corner to view the model in full lineage graph mode. - **Description** section — A [description of the model](/docs/build/documentation#adding-descriptions-to-your-project). - **Recent** section — Information on the last time the model ran, how long it ran for, whether the run was successful, the job ID, and the run ID. - - **Tests** section — [Tests](/docs/build/data-tests) for the model, including a status indicator for the latest test status. A :white_check_mark: denotes a passing test. + - **Tests** section — [Tests](/docs/build/data-tests) for the model, including a status indicator for the latest test status. A :white_check_mark: denotes a passing test. - **Details** section — Key properties like the model’s relation name (for example, how it’s represented and how you can query it in the data platform: `database.schema.identifier`); model governance attributes like access, group, and if contracted; and more. - **Relationships** section — The nodes the model **Depends On**, is **Referenced by**, and (if applicable) is **Used by** for projects that have declared the models' project as a dependency. - **Code** tab — The source code and compiled code for the model. -- **Columns** tab — The available columns in the model. This tab also shows tests results (if any) that you can select to view the test's details page. A :white_check_mark: denotes a passing test. To filter the columns in the resource, you can use the search bar that's located at the top of the columns view. +- **Columns** tab — The available columns in the model. This tab also shows tests results (if any) that you can select to view the test's details page. A :white_check_mark: denotes a passing test. To filter the columns in the resource, you can use the search bar that's located at the top of the columns view. -- **Status bar** (below the page title) — Information on the last time the exposure was updated. +- **Status bar** (below the page title) — Information on the last time the exposure was updated. - **Data health signals** — [Data health signals](/docs/collaborate/data-health-signals) offer a quick, at-a-glance view of data health. These icons indicate whether a resource is Healthy, Caution, or Degraded. Hover over an icon to view detailed information about the exposure's health. - **General** tab includes: - **Data health** — The status on data freshness and data quality. @@ -226,7 +226,7 @@ In the upper right corner of the resource details page, you can: - **Lineage** graph — The exposure’s lineage graph. Click the **Expand** icon in the graph's upper right corner to view the exposure in full lineage graph mode. Integrates natively with Tableau and auto-generates downstream lineage. - **Description** section — A description of the exposure. - **Details** section — Details like exposure type, maturity, owner information, and more. - - **Relationships** section — The nodes the exposure **Depends On**. + - **Relationships** section — The nodes the exposure **Depends On**. @@ -252,7 +252,7 @@ Example of the Tests view: -- **Status bar** (below the page title) — Information on the last time the source was updated and the number of tables the source uses. +- **Status bar** (below the page title) — Information on the last time the source was updated and the number of tables the source uses. - **Data health signals** — [Data health signals](/docs/collaborate/data-health-signals) offer a quick, at-a-glance view of data health. These icons indicate whether a resource is Healthy, Caution, or Degraded. Hover over an icon to view detailed information about the source's health. - **General** tab includes: - **Lineage** graph — The source’s lineage graph that you can interact with. The graph includes one upstream node and one downstream node from the source. Click the Expand icon in the graph's upper right corner to view the source in full lineage graph mode. @@ -277,13 +277,13 @@ Example of the details view for the model `customers`:
## Related content -- [Enterprise permissions](/docs/cloud/manage-access/enterprise-permissions) +- [Enterprise permissions](/docs/cloud/manage-access/enterprise-permissions) - [About model governance](/docs/collaborate/govern/about-model-governance) - Blog on [What is data mesh?](https://www.getdbt.com/blog/what-is-data-mesh-the-definition-and-importance-of-data-mesh) diff --git a/website/docs/docs/dbt-versions/release-notes/99-dbt-cloud-changelog-2019-2020.md b/website/docs/docs/dbt-versions/release-notes/99-dbt-cloud-changelog-2019-2020.md index 32a33d95301..721cf5e2d65 100644 --- a/website/docs/docs/dbt-versions/release-notes/99-dbt-cloud-changelog-2019-2020.md +++ b/website/docs/docs/dbt-versions/release-notes/99-dbt-cloud-changelog-2019-2020.md @@ -27,7 +27,7 @@ and adds two new permission sets for Enterprise acccounts. ## dbt Cloud v1.1.15 (December 10, 2020) -Lots of great stuff to confer about this go-round: things really coalesced this week! Lots of excitement around adding Spark to the connection family, as well as knocking out some longstanding bugs. +Lots of great stuff to confer about this go-round: things really coalesced this week! Lots of excitement around adding Spark to the connection family, as well as knocking out some longstanding bugs. #### Enhancements @@ -45,7 +45,7 @@ Lots of great stuff to confer about this go-round: things really coalesced this ## dbt Cloud v1.1.14 (November 25, 2020) -This release adds a few new pieces of connective tissue, notably OAuth for BigQuery and SparkAdapter work. There are also some quality of life improvements and investments for the future, focused on our beloved IDE users, and some improved piping for observability into log management and API usage. +This release adds a few new pieces of connective tissue, notably OAuth for BigQuery and SparkAdapter work. There are also some quality of life improvements and investments for the future, focused on our beloved IDE users, and some improved piping for observability into log management and API usage. #### Enhancements @@ -712,7 +712,7 @@ These fields need to be specified for your instance of dbt Cloud to function pro - Fix console warning presented when updating React state from unmounted component - Fix issue where closed tabs would continue to be shown, though the content was removed correctly - Fix issue that prevented opening an adjacent tab when a tab was closed -- Fix issue creating BigQuery connections causing the the account connections list to not load correctly. +- Fix issue creating BigQuery connections causing the account connections list to not load correctly. - Fix for locked accounts that have downgraded to the developer plan at trial end - Fix for not properly showing server error messages on the user invite page diff --git a/website/docs/reference/dbt-jinja-functions/run_started_at.md b/website/docs/reference/dbt-jinja-functions/run_started_at.md index 9dfc83ec56a..b8a25d8b80b 100644 --- a/website/docs/reference/dbt-jinja-functions/run_started_at.md +++ b/website/docs/reference/dbt-jinja-functions/run_started_at.md @@ -7,7 +7,7 @@ description: "Use `run_started_at` to output the timestamp the run started." `run_started_at` outputs the timestamp that this run started, e.g. `2017-04-21 01:23:45.678`. -The `run_started_at` variable is a Python `datetime` object. As of 0.9.1, the timezone of this variable +The `run_started_at` variable is a Python `datetime` object. As of 0.9.1, the timezone of this variable defaults to UTC. @@ -15,20 +15,20 @@ The `run_started_at` variable is a Python `datetime` object. As of 0.9.1, the ti ```sql select '{{ run_started_at.strftime("%Y-%m-%d") }}' as date_day - + from ... ``` -To modify the timezone of this variable, use the the `pytz` module: +To modify the timezone of this variable, use the `pytz` module: ```sql select '{{ run_started_at.astimezone(modules.pytz.timezone("America/New_York")) }}' as run_started_est - + from ... ``` diff --git a/website/docs/reference/resource-configs/bigquery-configs.md b/website/docs/reference/resource-configs/bigquery-configs.md index c912bca0688..f3bdc08ea34 100644 --- a/website/docs/reference/resource-configs/bigquery-configs.md +++ b/website/docs/reference/resource-configs/bigquery-configs.md @@ -14,7 +14,7 @@ To-do: - `schema` is interchangeable with the BigQuery concept `dataset` - `database` is interchangeable with the BigQuery concept of `project` -For our reference documentation, you can declare `project` in place of `database.` +For our reference documentation, you can declare `project` in place of `database.` This will allow you to read and write from multiple BigQuery projects. Same for `dataset`. ## Using table partitioning and clustering @@ -335,7 +335,7 @@ models: dbt supports the specification of BigQuery labels for the tables and views that it creates. These labels can be specified using the `labels` model config. The `labels` config can be provided in a model config, or in the `dbt_project.yml` file, as shown below. - + BigQuery key-value pair entries for labels larger than 63 characters are truncated. **Configuring labels in a model file** @@ -393,9 +393,9 @@ select * from {{ ref('another_model') }} -You can create a new label with no value or remove a value from an existing label key. +You can create a new label with no value or remove a value from an existing label key. -A label with a key that has an empty value can also be [referred](https://cloud.google.com/bigquery/docs/adding-labels#adding_a_label_without_a_value) to as a tag in BigQuery. However, this should not be confused with a [tag resource](https://cloud.google.com/bigquery/docs/tags), which conditionally applies IAM policies to BigQuery tables and datasets. Find out more in [labels and tags](https://cloud.google.com/resource-manager/docs/tags/tags-overview). +A label with a key that has an empty value can also be [referred](https://cloud.google.com/bigquery/docs/adding-labels#adding_a_label_without_a_value) to as a tag in BigQuery. However, this should not be confused with a [tag resource](https://cloud.google.com/bigquery/docs/tags), which conditionally applies IAM policies to BigQuery tables and datasets. Find out more in [labels and tags](https://cloud.google.com/resource-manager/docs/tags/tags-overview). Currently, it's not possible to apply IAM tags in BigQuery, however, you can weigh in by upvoting [GitHub issue 1134](https://github.com/dbt-labs/dbt-bigquery/issues/1134). @@ -551,7 +551,7 @@ _today_ and _yesterday_ every day that it is run. It is the fastest and cheapest way to incrementally update a table using dbt. If we wanted this to run more dynamically— let’s say, always for the past 3 days—we could leverage dbt’s baked-in [datetime macros](https://github.com/dbt-labs/dbt-core/blob/dev/octavius-catto/core/dbt/include/global_project/macros/etc/datetime.sql) and write a few of our own. -Think of this as "full control" mode. You must ensure that expressions or literal values in the the `partitions` config have proper quoting when templated, and that they match the `partition_by.data_type` (`timestamp`, `datetime`, `date`, or `int64`). Otherwise, the filter in the incremental `merge` statement will raise an error. +Think of this as "full control" mode. You must ensure that expressions or literal values in the `partitions` config have proper quoting when templated, and that they match the `partition_by.data_type` (`timestamp`, `datetime`, `date`, or `int64`). Otherwise, the filter in the incremental `merge` statement will raise an error. #### Dynamic partitions