From 6824b28e91fdd521f3a2b7b9fee20ca36f9ecc70 Mon Sep 17 00:00:00 2001 From: "albert.franzi" <3647015+afranzi@users.noreply.github.com> Date: Fri, 2 Feb 2024 18:01:32 +0100 Subject: [PATCH] fix: :wrench - Refine documentation --- docs/index.md | 22 ++++++++++++++-------- docs/setup.md | 43 +++++++++++++++++++++++++++---------------- 2 files changed, 41 insertions(+), 24 deletions(-) diff --git a/docs/index.md b/docs/index.md index b384ef4..1110ee9 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,25 +1,31 @@ # Introduction -This guide outlines how to override the default AWS connections backend in Apache Airflow with a custom backend. -This enables Airflow to access secrets stored in other AWS accounts, in addition to the current one, -leveraging [AWS Secrets Manager](https://aws.amazon.com/es/secrets-manager/). +This comprehensive guide details the steps to enhance Apache Airflow's default AWS connections backend by integrating a +custom backend. This advanced setup allows Airflow to seamlessly access secrets not only within its AWS account but also +across multiple AWS accounts, utilizing [AWS Secrets Manager](https://aws.amazon.com/es/secrets-manager/). +The integration enhances security and flexibility, enabling centralized secret management across diverse cloud +environments. ## Overview -By default, Airflow uses the `airflow.secrets` base class -for [secrets backends](https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/secrets-backend/index.html). -To access secrets across different AWS accounts, we implemented a custom backend that extends this functionality. +Apache Airflow's default configuration utilizes the `airflow.secrets base class for managing secrets, such as database +credentials and API keys.
![Image title](./images/aws-secrets-flow.jpg){ width="500" }
SecretsManagerBackend Flow
+Our approach extends this functionality with +a [custom backend](https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/secrets-backend/index.html), +specifically designed to retrieve secrets from AWS Secrets Manager across different AWS accounts and regions. +This solution facilitates secure and efficient secret management for complex cloud architectures. !!! note "SecretsManagerBackend class" - In short, we extended + In essence, we enhance the [SecretsManagerBackend](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/secrets-backends/aws-secrets-manager.html) - class with cross-account and cross-region secrets access. + to enable cross-account and cross-region secrets access, thereby providing a more versatile and secure secrets + management strategy.
![Image title](./images/aws-shared-secrets-flow.jpg){ width="600" } diff --git a/docs/setup.md b/docs/setup.md index 81bb8e3..47103b0 100644 --- a/docs/setup.md +++ b/docs/setup.md @@ -1,25 +1,28 @@ ## Prerequisites -- Apache Airflow 2.x -- Access to the AWS accounts from which you want to fetch secrets +- **Apache Airflow 2.x:** Ensure Airflow is updated to at least version 2.0. +- **AWS Account Access:** You need access to the AWS accounts from which secrets will be fetched _( See: [Permissions to AWS Secrets Manager secrets for users in a different account](https://docs.aws.amazon.com/secretsmanager/latest/userguide/auth-and-access_examples_cross.html))_ -- Permissions to create and manage secrets in AWS Secrets Manager -- Access to edit your Airflow Helm properties. +- **Permissions**: Adequate permissions to manage secrets within AWS Secrets Manager and configure IAM policies. +- **Helm:** Familiarity with Helm for deploying and managing Kubernetes applications, as this guide uses the Airflow + Helm Chart. ## Helm configuration -The documentation is based on [Airflow Helm Chart (Users Community)](https://airflow-helm.github.io/charts/), but it -should apply to other existing Airflow Helms, since the configuration would be handled via ENV vars or the config file. +This documentation leverages the [Airflow Helm Chart (Users Community)](https://airflow-helm.github.io/charts/). The +instructions should be universally applicable across different Helm deployments, thanks to the flexibility of +environment variables and configuration files. -!!! note extraPipPackages - Add the **airflow-aws-shared-secrets** package in the `extraPipPackages` section, so the library is deployed into - the airflow workers pods. +!!! note "extraPipPackages" + Ensure the **airflow-aws-shared-secrets** package is included in the `extraPipPackages` section of your Helm values. + This ensures the custom library is deployed into the Airflow worker pods, enabling them to interact with the custom + backend. ## Configure Airflow to use our Custom Backend -To use the AWs Shared Secrets Backed in Airflow, we must update the airflow.cfg file or set the corresponding -environment variable: +To activate the AWS Shared Secrets Backend in Airflow, adjust either the `airflow.cfg` file directly or set the +appropriate environment variables: Edit airflow.cfg: @@ -38,18 +41,26 @@ AIRFLOW__SECRETS__BACKEND_KWARGS: '{"connections_prefix": "airflow/connections/" ## Backend properties -We expect the following extra properties to be defined within the `backend_kwargs` in addition to +The custom backend expects additional properties within `backend_kwargs` enhancing functionality beyond the native capabilities in addition to the [native ones](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/secrets-backends/aws-secrets-manager.html#aws-secrets-manager-backend). -- shared_account: Account ID from the aws-account you are sharing the main secrets. -- aws_region: The AWs Region from where the secrets are being stored in. +- shared_account: Specifies the AWS account ID where the primary secrets are stored. This facilitates cross-account secret access. +- aws_region: Defines the AWS region of the secrets manager, ensuring the backend can retrieve secrets from the specified geographical location. -!!! note Conclusion - By following these steps, you've successfully overridden the default AWS connections backend in Airflow with a custom one that allows accessing secrets from other AWS accounts. +!!! info "Conclusion" + By following these steps, you've successfully overridden the default AWS connections backend in Airflow with a custom + one that allows accessing secrets from other AWS accounts. This setup enhances your Airflow project's flexibility and security when managing cross-account AWS resources. +## Implementing Best Practices +When configuring and using the custom backend, adhere to the following best practices for security and efficiency: + +- **Minimal IAM Permissions:** Assign only the necessary permissions to the IAM roles used by Airflow, following the principle of least privilege. +- **Secure Secret Storage:** Ensure that all secrets stored in AWS Secrets Manager are encrypted at rest using keys managed by AWS KMS. +- **Regular Audits:** Periodically review AWS access logs and Airflow access patterns to ensure compliance with security policies. ## Specs Example + ```yaml apiVersion: v2 kind: HelmRelease