From 593fecae0362b205ca56b7c115060287b61e91ef Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 17 Dec 2024 18:32:40 -0500 Subject: [PATCH] Private packages beta (#6675) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## What are you changing in this pull request and why? Beta docs for Private packages ## Checklist - [ ] I have reviewed the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ ] The topic I'm writing about is for specific dbt version(s) and I have versioned it according to the [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and/or [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content) guidelines. - [ ] I have added checklist item(s) to this list for anything anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." - [ ] The content in this PR requires a dbt release note, so I added one to the [release notes page](https://docs.getdbt.com/docs/dbt-versions/dbt-cloud-release-notes). --- 🚀 Deployment available! Here are the direct links to the updated files: - https://docs-getdbt-com-git-private-packages-dbt-labs.vercel.app/docs/build/packages --------- Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Co-authored-by: Leona B. Campbell <3880403+runleonarun@users.noreply.github.com> --- website/docs/docs/build/packages.md | 63 +++++++++++++++++++++++++++-- 1 file changed, 59 insertions(+), 4 deletions(-) diff --git a/website/docs/docs/build/packages.md b/website/docs/docs/build/packages.md index 9ba4ceeaff5..82ba2c3d74c 100644 --- a/website/docs/docs/build/packages.md +++ b/website/docs/docs/build/packages.md @@ -157,9 +157,57 @@ packages: Where `name: 'dbt_utils'` specifies the subfolder of `dbt_packages` that's created for the package source code to be installed within. -### Private packages +## Private packages -#### SSH Key Method (Command Line only) +### Native private packages + +dbt Cloud supports private packages from [supported](#prerequisites) Git repos leveraging an exisiting [configuration](/docs/cloud/git/git-configuration-in-dbt-cloud) in your environment. Previously, you had to configure a [token](#git-token-method) to retrieve packages from your private repos. + +#### Prerequisites + +To use native private packages, you must have one of the following Git providers configured in the **Integrations** section of your **Account settings**: +- [GitHub](/docs/cloud/git/connect-github) +- [Azure DevOps](/docs/cloud/git/connect-azure-devops) +- Support for GitLab is coming soon. + + +#### Configuration + +Use the `private` key in your `packages.yml` or `dependencies.yml` to clone package repos using your existing dbt Cloud Git integration without having to provision an access token or create a dbt Cloud environment variable: + + + +```yaml +packages: + - private: dbt-labs/awesome_repo + - package: normal packages + + [...] +``` + + + +You can pin private packages similar to regular dbt packages: + +```yaml +packages: + - private: dbt-labs/awesome_repo + revision: "0.9.5" # Pin to a tag, branch, or complete 40-character commit hash + +``` + +If you are using multiple Git integrations, disambiguate by adding the provider key: + +```yaml +packages: + - private: dbt-labs/awesome_repo + provider: "github" # GitHub and Azure are currently supported. GitLab is coming soon. + +``` + +With this method, you can retrieve private packages from an integrated Git provider without any additional steps to connect. + +### SSH key method (command line only) If you're using the Command Line, private packages can be cloned via SSH and an SSH key. When you use SSH keys to authenticate to your git remote server, you don’t need to supply your username and password each time. Read more about SSH keys, how to generate them, and how to add them to your git provider here: [Github](https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh) and [GitLab](https://docs.gitlab.com/ee/user/ssh.html). @@ -177,7 +225,14 @@ packages: If you're using dbt Cloud, the SSH key method will not work, but you can use the [HTTPS Git Token Method](https://docs.getdbt.com/docs/build/packages#git-token-method). -#### Git token method +### Git token method + +:::note + +dbt Cloud has [native support](#native-private-packages) for Git hosted private packages with GitHub and Azure DevOps (GitLab coming soon). If you are using a supported [integrated Git environment](/docs/cloud/git/git-configuration-in-dbt-cloud), you no longer need to configure Git tokens to retrieve private packages. + +::: + This method allows the user to clone via HTTPS by passing in a git token via an environment variable. Be careful of the expiration date of any token you use, as an expired token could cause a scheduled run to fail. Additionally, user tokens can create a challenge if the user ever loses access to a specific repo. @@ -246,7 +301,7 @@ Read more about creating a Personal Access Token [here](https://confluence.atlas -#### Configure subdirectory for packaged projects +## Configure subdirectory for packaged projects In general, dbt expects `dbt_project.yml` to be located as a top-level file in a package. If the packaged project is instead nested in a subdirectory—perhaps within a much larger mono repo—you can optionally specify the folder path as `subdirectory`. dbt will attempt a [sparse checkout](https://git-scm.com/docs/git-sparse-checkout) of just the files located within that subdirectory. Note that you must be using a recent version of `git` (`>=2.26.0`).