Skip to content

Commit

Permalink
Merge branch 'current' into shutdown-final-terms
Browse files Browse the repository at this point in the history
  • Loading branch information
JKarlavige authored Jan 17, 2025
2 parents 6bf472a + f7b1819 commit 7f023c0
Show file tree
Hide file tree
Showing 39 changed files with 721 additions and 612 deletions.
24 changes: 10 additions & 14 deletions website/docs/docs/build/enhance-your-code.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,21 +7,17 @@ pagination_prev: null

<div className="grid--2-col">

<Card
title="Project variables"
body="Learn how to use project variables to provide data to models for compilation."
link="/docs/build/project-variables"
icon="dbt-bit"/>

<Card
title="Environment variables"
body="Learn how you can use environment variables to customize the behavior of a dbt project."
link="/docs/build/environment-variables"
icon="dbt-bit"/>

</div>
<br />
<div className="grid--2-col">
<Card
title="Hooks and operations"
body="Learn how to use hooks to trigger actions and operations to invoke macros."
link="/docs/build/hooks-operations"
icon="dbt-bit"/>

<Card
title="Packages"
Expand All @@ -30,9 +26,9 @@ pagination_prev: null
icon="dbt-bit"/>

<Card
title="Hooks and operations"
body="Learn how to use hooks to trigger actions and operations to invoke macros."
link="/docs/build/hooks-operations"
icon="dbt-bit"/>
title="Project variables"
body="Learn how to use project variables to provide data to models for compilation."
link="/docs/build/project-variables"
icon="dbt-bit"/>

</div>
</div>
17 changes: 15 additions & 2 deletions website/docs/docs/build/incremental-microbatch.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,14 +29,27 @@ Microbatch is an incremental strategy designed for large time-series datasets:

- Note, microbatch might not be the best strategy for all use cases. Consider other strategies for use cases such as not having a reliable `event_time` column or if you want more control over the incremental logic. Read more in [How `microbatch` compares to other incremental strategies](#how-microbatch-compares-to-other-incremental-strategies).

### How microbatch works
## How microbatch works

When dbt runs a microbatch model — whether for the first time, during incremental runs, or in specified backfills — it will split the processing into multiple queries (or "batches"), based on the `event_time` and `batch_size` you configure.

Each "batch" corresponds to a single bounded time period (by default, a single day of data). Where other incremental strategies operate only on "old" and "new" data, microbatch models treat every batch as an atomic unit that can be built or replaced on its own. Each batch is independent and <Term id="idempotent" />.

This is a powerful abstraction that makes it possible for dbt to run batches [separately](#backfills), concurrently, and [retry](#retry) them independently.

### Adapter-specific behavior

dbt's microbatch strategy uses the most efficient mechanism available for "full batch" replacement on each adapter. This can vary depending on the adapter:

- `dbt-postgres`: Uses the `merge` strategy, which performs "update" or "insert" operations.
- `dbt-redshift`: Uses the `delete+insert` strategy, which "inserts" or "replaces."
- `dbt-snowflake`: Uses the `delete+insert` strategy, which "inserts" or "replaces."
- `dbt-bigquery`: Uses the `insert_overwrite` strategy, which "inserts" or "replaces."
- `dbt-spark`: Uses the `insert_overwrite` strategy, which "inserts" or "replaces."
- `dbt-databricks`: Uses the `replace_where` strategy, which "inserts" or "replaces."

Check out the [supported incremental strategies by adapter](/docs/build/incremental-strategy#supported-incremental-strategies-by-adapter) for more info.

## Example

A `sessions` model aggregates and enriches data that comes from two other models:
Expand Down Expand Up @@ -170,7 +183,7 @@ customers as (

</Tabs>

dbt will instruct the data platform to take the result of each batch query and insert, update, or replace the contents of the `analytics.sessions` table for the same day of data. To perform this operation, dbt will use the most efficient atomic mechanism for "full batch" replacement that is available on each data platform.
dbt will instruct the data platform to take the result of each batch query and [insert, update, or replace](#adapter-specific-behavior) the contents of the `analytics.sessions` table for the same day of data. To perform this operation, dbt will use the most efficient atomic mechanism for "full batch" replacement that is available on each data platform. For details, see [How microbatch works](#how-microbatch-works).

It does not matter whether the table already contains data for that day. Given the same input data, the resulting table is the same no matter how many times a batch is reprocessed.

Expand Down
1 change: 1 addition & 0 deletions website/docs/docs/build/incremental-strategy.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
---
title: "About incremental strategy"
sidebar_label: "About incremental strategy"
description: "Learn about the various ways (strategies) to implement incremental materializations."
id: "incremental-strategy"
---
Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/cloud/about-cloud-develop-defer.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Using defer in dbt Cloud
id: about-cloud-develop-defer
description: "Learn how to leverage defer to prod when developing with dbt Cloud."
sidebar_label: "Using defer in dbt Cloud"
sidebar_label: "Defer in dbt Cloud"
pagination_next: "docs/cloud/cloud-cli-installation"
---

Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/cloud/dbt-cloud-ide/lint-format.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ To configure your own linting rules:

1. Create a new file in the root project directory (the parent or top-level directory for your files). Note: The root project directory is the directory where your `dbt_project.yml` file resides.
2. Name the file `.sqlfluff` (make sure you add the `.` before `sqlfluff`).
3. [Create](https://docs.sqlfluff.com/en/stable/configuration.html#new-project-configuration) and add your custom config code.
3. [Create](https://docs.sqlfluff.com/en/stable/configuration/setting_configuration.html#new-project-configuration) and add your custom config code.
4. Save and commit your changes.
5. Restart the IDE.
6. Test it out and happy linting!
Expand Down
Loading

0 comments on commit 7f023c0

Please sign in to comment.