Skip to content

Commit

Permalink
Add terraform example for Azure pipeline (close #65)
Browse files Browse the repository at this point in the history
  • Loading branch information
jbeemster authored and stanch committed Aug 3, 2023
1 parent 3f195c3 commit c16eafa
Show file tree
Hide file tree
Showing 21 changed files with 997 additions and 223 deletions.
252 changes: 51 additions & 201 deletions LICENSE

Large diffs are not rendered by default.

45 changes: 23 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,18 @@ These examples cover deploying an Iglu Server, for hosting your schemas, and a S

## Supported Platforms

| Tool | Cloud | Components | Status | Deployment Summary |
|------------|-------|-------------------------|---------------------------------------------|-------------------------------------------|
| Terraform | AWS | Iglu Server | [Published](terraform/aws/iglu_server) | |
| Terraform | AWS | Pipeline (PostgreSQL) | [Published](terraform/aws/pipeline) | [AWS PostgreSQL Summary][deploypgsum-aws] |
| Terraform | AWS | Pipeline (Snowflake) | [Published](terraform/aws/snowflake) | [AWS Snowflake Summary][deploysfsum-aws] |
| Terraform | GCP | Iglu Server | [Published](terraform/gcp/iglu_server) | |
| Terraform | GCP | Pipeline (PostgreSQL) | [Published](terraform/gcp/pipeline) | [GCP PostgreSQL Summary][deploysum-gcp] |
| Terraform | GCP | Pipeline (BigQuery) | [Published](terraform/gcp/pipeline) | [GCP BigQuery Summary][deploysum-gcp] |
| Tool | Cloud | Components | Status | Deployment Summary |
|------------|-------|-------------------------|---------------------------------------------|----------------------------------------------|
| Terraform | AWS | Iglu Server | [Published](terraform/aws/iglu_server) | |
| Terraform | AWS | Pipeline (PostgreSQL) | [Published](terraform/aws/pipeline) | [AWS PostgreSQL Summary][deploypgsum-aws] |
| Terraform | AWS | Pipeline (Snowflake) | [Published](terraform/aws/pipeline) | [AWS Snowflake Summary][deploysfsum-aws] |
| Terraform | AWS | Pipeline (Redshift) | [Published](terraform/aws/pipeline) | [AWS Redshift Summary][deployrssum-aws] |
| Terraform | AWS | Pipeline (Databricks) | [Published](terraform/aws/pipeline) | [AWS Databricks Summary][deploydbsum-aws] |
| Terraform | GCP | Iglu Server | [Published](terraform/gcp/iglu_server) | |
| Terraform | GCP | Pipeline (PostgreSQL) | [Published](terraform/gcp/pipeline) | [GCP PostgreSQL Summary][deploypgsum-gcp] |
| Terraform | GCP | Pipeline (BigQuery) | [Published](terraform/gcp/pipeline) | [GCP BigQuery Summary][deploybqsum-gcp] |
| Terraform | Azure | Iglu Server | [Published](terraform/azure/iglu_server) | |
| Terraform | Azure | Pipeline (Snowflake) | [Published](terraform/azure/pipeline) | [Azure Snowflake Summary][deploysfsum-azure] |

## Documentation

Expand Down Expand Up @@ -45,16 +49,9 @@ terraform-docs -c .terraform-docs.yml terraform/gcp/pipeline/secure

## Copyright and license

The Snowplow Quick start examples are copyright 2022 Snowplow Analytics Ltd.
Copyright (c) 2022-present Snowplow Analytics Ltd. All rights reserved.

Licensed under the **[Apache License, Version 2.0][license]** (the "License");
you may not use this software except in compliance with the License.

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Licensed under the [Snowplow Community License](https://docs.snowplow.io/community-license-1.0). _(If you are uncertain how it applies to your use case, check our answers to [frequently asked questions](https://docs.snowplow.io/docs/contributing/community-license-faq/).)_

[install-image]: https://d3i6fms1cm1j0i.cloudfront.net/github/images/techdocs.png
[deploy-image]: https://d3i6fms1cm1j0i.cloudfront.net/github/images/setup.png
Expand All @@ -63,12 +60,16 @@ limitations under the License.
[installguide]: https://docs.snowplow.io/docs/open-source-quick-start/
[faq]: https://docs.snowplow.io/docs/open-source-quick-start/quick-start-faqs/

[deploypgsum-aws]: https://docs.snowplow.io/docs/open-source-quick-start/quick-start-installation-guide-on-aws/summary-of-what-you-have-deployed/aws-and-postgres/
[deploysfsum-aws]: https://docs.snowplow.io/docs/open-source-quick-start/quick-start-installation-guide-on-aws/summary-of-what-you-have-deployed/aws-and-snowflake/
[deploysum-gcp]: https://docs.snowplow.io/docs/open-source-quick-start/quick-start-installation-guide-on-gcp/summary-of-what-you-have-deployed/
[deploypgsum-aws]: https://docs.snowplow.io/docs/getting-started-on-snowplow-open-source/what-is-deployed/?warehouse=postgres&cloud=aws
[deploysfsum-aws]: https://docs.snowplow.io/docs/getting-started-on-snowplow-open-source/what-is-deployed/?warehouse=snowflake&cloud=aws
[deployrssum-aws]: https://docs.snowplow.io/docs/getting-started-on-snowplow-open-source/what-is-deployed/?warehouse=redshift&cloud=aws
[deploydbsum-aws]: https://docs.snowplow.io/docs/getting-started-on-snowplow-open-source/what-is-deployed/?warehouse=databricks&cloud=aws
[deploypgsum-gcp]: https://docs.snowplow.io/docs/getting-started-on-snowplow-open-source/what-is-deployed/?warehouse=postgres&cloud=gcp
[deploybqsum-gcp]: https://docs.snowplow.io/docs/getting-started-on-snowplow-open-source/what-is-deployed/?warehouse=bigquery&cloud=gcp
[deploysfsum-azure]: https://docs.snowplow.io/docs/getting-started-on-snowplow-open-source/what-is-deployed/?warehouse=snowflake&cloud=azure

[license-image]: https://img.shields.io/badge/license-Apache--2-blue.svg?style=flat
[license]: https://www.apache.org/licenses/LICENSE-2.0
[license]: https://docs.snowplow.io/docs/contributing/community-license-faq/
[license-image]: https://img.shields.io/badge/license-Snowplow--Community-blue.svg?style=flat

[discourse-image]: https://img.shields.io/discourse/posts?server=https%3A%2F%2Fdiscourse.snowplow.io%2F
[discourse]: http://discourse.snowplow.io/
Expand Down
43 changes: 43 additions & 0 deletions terraform/azure/base/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
## Requirements

| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0.0 |
| <a name="requirement_azurerm"></a> [azurerm](#requirement\_azurerm) | >= 3.58.0 |

## Providers

| Name | Version |
|------|---------|
| <a name="provider_azurerm"></a> [azurerm](#provider\_azurerm) | >= 3.58.0 |

## Modules

| Name | Source | Version |
|------|--------|---------|
| <a name="module_vnet"></a> [vnet](#module\_vnet) | snowplow-devops/vnet/azurerm | 0.1.2 |

## Resources

| Name | Type |
|------|------|
| [azurerm_resource_group.rg](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/resource_group) | resource |

## Inputs

| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| <a name="input_location"></a> [location](#input\_location) | The location in which all resources will be created (e.g. australiaeast) | `string` | n/a | yes |
| <a name="input_prefix"></a> [prefix](#input\_prefix) | Will be prefixed to all resource names. Use to easily identify the resources created | `string` | n/a | yes |
| <a name="input_tags"></a> [tags](#input\_tags) | The tags to append to the resources in this module | `map(string)` | `{}` | no |

## Outputs

| Name | Description |
|------|-------------|
| <a name="output_resource_group_location"></a> [resource\_group\_location](#output\_resource\_group\_location) | The location of the resource group |
| <a name="output_resource_group_name"></a> [resource\_group\_name](#output\_resource\_group\_name) | The name of the resource group |
| <a name="output_vnet_id"></a> [vnet\_id](#output\_vnet\_id) | The ID of the vNet |
| <a name="output_vnet_name"></a> [vnet\_name](#output\_vnet\_name) | The name of the vNet |
| <a name="output_vnet_subnets"></a> [vnet\_subnets](#output\_vnet\_subnets) | The IDs of subnets created inside the vNet |
| <a name="output_vnet_subnets_name_id"></a> [vnet\_subnets\_name\_id](#output\_vnet\_subnets\_name\_id) | Can be queried subnet ID by subnet name by using lookup(module.vnet.vnet\_subnets\_name\_id, subnet1) |
18 changes: 18 additions & 0 deletions terraform/azure/base/main.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
resource "azurerm_resource_group" "rg" {
name = "${var.prefix}-rg"
location = var.location

tags = var.tags
}

module "vnet" {
source = "snowplow-devops/vnet/azurerm"
version = "0.1.2"

name = "${var.prefix}-vnet"
resource_group_name = azurerm_resource_group.rg.name

tags = var.tags

depends_on = [azurerm_resource_group.rg]
}
29 changes: 29 additions & 0 deletions terraform/azure/base/outputs.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
output "resource_group_name" {
description = "The name of the resource group"
value = azurerm_resource_group.rg.name
}

output "resource_group_location" {
description = "The location of the resource group"
value = azurerm_resource_group.rg.location
}

output "vnet_id" {
description = "The ID of the vNet"
value = module.vnet.vnet_id
}

output "vnet_name" {
description = "The name of the vNet"
value = module.vnet.vnet_name
}

output "vnet_subnets" {
description = "The IDs of subnets created inside the vNet"
value = module.vnet.vnet_subnets
}

output "vnet_subnets_name_id" {
description = "Can be queried subnet ID by subnet name by using lookup(module.vnet.vnet_subnets_name_id, subnet1)"
value = module.vnet.vnet_subnets_name_id
}
9 changes: 9 additions & 0 deletions terraform/azure/base/terraform.tfvars
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Will be prefixed to all resource names
# Use this to easily identify the resources created and provide entropy for subsequent environments
prefix = "snowplow"

# The location to create resources in: https://azure.microsoft.com/en-au/explore/global-infrastructure/geographies/#overview
location = "<ADD_ME>"

# Extra Tags to append to created resources (optional)
tags = {}
15 changes: 15 additions & 0 deletions terraform/azure/base/variables.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
variable "prefix" {
description = "Will be prefixed to all resource names. Use to easily identify the resources created"
type = string
}

variable "location" {
description = "The location in which all resources will be created (e.g. australiaeast)"
type = string
}

variable "tags" {
description = "The tags to append to the resources in this module"
default = {}
type = map(string)
}
14 changes: 14 additions & 0 deletions terraform/azure/base/versions.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
terraform {
required_version = ">= 1.0.0"

required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = ">= 3.58.0"
}
}
}

provider "azurerm" {
features {}
}
49 changes: 49 additions & 0 deletions terraform/azure/iglu_server/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
## Requirements

| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0.0 |
| <a name="requirement_azurerm"></a> [azurerm](#requirement\_azurerm) | >= 3.58.0 |

## Providers

No providers.

## Modules

| Name | Source | Version |
|------|--------|---------|
| <a name="module_iglu_db"></a> [iglu\_db](#module\_iglu\_db) | snowplow-devops/postgresql-server/azurerm | 0.1.1 |
| <a name="module_iglu_lb"></a> [iglu\_lb](#module\_iglu\_lb) | snowplow-devops/lb/azurerm | 0.1.1 |
| <a name="module_iglu_server"></a> [iglu\_server](#module\_iglu\_server) | snowplow-devops/iglu-server-vmss/azurerm | 0.1.1 |

## Resources

No resources.

## Inputs

| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| <a name="input_iglu_db_name"></a> [iglu\_db\_name](#input\_iglu\_db\_name) | The name of the database to create | `string` | n/a | yes |
| <a name="input_iglu_db_password"></a> [iglu\_db\_password](#input\_iglu\_db\_password) | The password to use to connect to the database | `string` | n/a | yes |
| <a name="input_iglu_db_username"></a> [iglu\_db\_username](#input\_iglu\_db\_username) | The username to use to connect to the database | `string` | n/a | yes |
| <a name="input_iglu_super_api_key"></a> [iglu\_super\_api\_key](#input\_iglu\_super\_api\_key) | A UUIDv4 string to use as the master API key for Iglu Server management | `string` | n/a | yes |
| <a name="input_prefix"></a> [prefix](#input\_prefix) | Will be prefixed to all resource names. Use to easily identify the resources created | `string` | n/a | yes |
| <a name="input_resource_group_name"></a> [resource\_group\_name](#input\_resource\_group\_name) | The name of the resource group to deploy resources within | `string` | n/a | yes |
| <a name="input_ssh_ip_allowlist"></a> [ssh\_ip\_allowlist](#input\_ssh\_ip\_allowlist) | The list of CIDR ranges to allow SSH traffic from | `list(any)` | n/a | yes |
| <a name="input_ssh_public_key"></a> [ssh\_public\_key](#input\_ssh\_public\_key) | The SSH public key to use for the deployment | `string` | n/a | yes |
| <a name="input_subnet_id_lb"></a> [subnet\_id\_lb](#input\_subnet\_id\_lb) | The ID of the subnet to deploy the load balancer into (e.g. iglu-agw1) | `string` | n/a | yes |
| <a name="input_subnet_id_servers"></a> [subnet\_id\_servers](#input\_subnet\_id\_servers) | The ID of the subnet to deploy the servers into (e.g. iglu1) | `string` | n/a | yes |
| <a name="input_iglu_db_ip_allowlist"></a> [iglu\_db\_ip\_allowlist](#input\_iglu\_db\_ip\_allowlist) | An optional list of CIDR ranges to allow traffic from | `list(any)` | `[]` | no |
| <a name="input_ssl_information"></a> [ssl\_information](#input\_ssl\_information) | SSL certificate information to optionally bind to the load balancer | <pre>object({<br> enabled = bool<br> data = string<br> password = string<br> })</pre> | <pre>{<br> "data": "",<br> "enabled": false,<br> "password": ""<br>}</pre> | no |
| <a name="input_tags"></a> [tags](#input\_tags) | The tags to append to the resources in this module | `map(string)` | `{}` | no |
| <a name="input_telemetry_enabled"></a> [telemetry\_enabled](#input\_telemetry\_enabled) | Whether or not to send telemetry information back to Snowplow Analytics Ltd | `bool` | `true` | no |
| <a name="input_user_provided_id"></a> [user\_provided\_id](#input\_user\_provided\_id) | An optional unique identifier to identify the telemetry events emitted by this stack | `string` | `""` | no |

## Outputs

| Name | Description |
|------|-------------|
| <a name="output_iglu_server_lb_fqdn"></a> [iglu\_server\_lb\_fqdn](#output\_iglu\_server\_lb\_fqdn) | The load balancers fully-qualified-domain-name for the Iglu Server |
| <a name="output_iglu_server_lb_ip_address"></a> [iglu\_server\_lb\_ip\_address](#output\_iglu\_server\_lb\_ip\_address) | The load balancers IP address for the Iglu Server |
63 changes: 63 additions & 0 deletions terraform/azure/iglu_server/main.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
module "iglu_db" {
source = "snowplow-devops/postgresql-server/azurerm"
version = "0.1.1"

name = "${var.prefix}-iglu-db"
resource_group_name = var.resource_group_name

subnet_id = var.subnet_id_servers

additional_ip_allowlist = var.iglu_db_ip_allowlist

db_name = var.iglu_db_name
db_username = var.iglu_db_username
db_password = var.iglu_db_password

tags = var.tags
}

module "iglu_lb" {
source = "snowplow-devops/lb/azurerm"
version = "0.1.1"

name = "${var.prefix}-iglu-lb"
resource_group_name = var.resource_group_name
subnet_id = var.subnet_id_lb

probe_path = "/api/meta/health"

ssl_certificate_enabled = var.ssl_information.enabled
ssl_certificate_data = var.ssl_information.data
ssl_certificate_password = var.ssl_information.password

tags = var.tags
}

module "iglu_server" {
source = "snowplow-devops/iglu-server-vmss/azurerm"
version = "0.1.1"

name = "${var.prefix}-iglu-server"
resource_group_name = var.resource_group_name
subnet_id = var.subnet_id_servers

application_gateway_backend_address_pool_ids = [module.iglu_lb.agw_backend_address_pool_id]

ingress_port = module.iglu_lb.agw_backend_egress_port

ssh_public_key = var.ssh_public_key
ssh_ip_allowlist = var.ssh_ip_allowlist

db_name = module.iglu_db.db_name
db_host = module.iglu_db.db_host
db_port = module.iglu_db.db_port
db_username = module.iglu_db.db_username
db_password = module.iglu_db.db_password

super_api_key = var.iglu_super_api_key

telemetry_enabled = var.telemetry_enabled
user_provided_id = var.user_provided_id

tags = var.tags
}
9 changes: 9 additions & 0 deletions terraform/azure/iglu_server/outputs.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
output "iglu_server_lb_ip_address" {
description = "The load balancers IP address for the Iglu Server"
value = module.iglu_lb.ip_address
}

output "iglu_server_lb_fqdn" {
description = "The load balancers fully-qualified-domain-name for the Iglu Server"
value = module.iglu_lb.ip_address_fqdn
}
45 changes: 45 additions & 0 deletions terraform/azure/iglu_server/terraform.tfvars
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Will be prefixed to all resource names
# Use this to easily identify the resources created and provide entropy for subsequent environments
prefix = "snowplow"

# The name of the resource group to deploy Iglu into
resource_group_name = "<ADD_ME>"

# ID of the dedicated subnet to deploy the load balancer into
subnet_id_lb = "<SET_ME>"

# ID of the subnet to deploy the actual Iglu Server application into
subnet_id_servers = "<SET_ME>"

# Update this to _your_ IP Address
ssh_ip_allowlist = ["999.999.999.999/32"]
# Generate a new SSH key locally with `ssh-keygen`
# ssh-keygen -t rsa -b 4096
ssh_public_key = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQA0jSi9//bRsHW4M6czodTs6smCXsxZ0gijzth0aBmycE= [email protected]"

# Iglu Server settings
iglu_db_name = "iglu"
iglu_db_username = "iglu"
# Change and keep this secret!
iglu_db_password = "<PASSWORD>"

# Used for API actions on the Iglu Server
# Change this to a new UUID and keep it secret!
iglu_super_api_key = "00000000-0000-0000-0000-000000000000"

# NOTE: To push schemas to your Iglu Server, you can use igluctl
# See the docs: https://docs.snowplow.io/docs/understanding-tracking-design/managing-your-data-structures/iglu/

# Telemetry principles: https://docs.snowplow.io/docs/getting-started-on-snowplow-open-source/telemetry/
user_provided_id = ""
telemetry_enabled = true

# SSL Configuration (optional)
ssl_information = {
password = ""
data = ""
enabled = false
}

# Extra Tags to append to created resources (optional)
tags = {}
Loading

0 comments on commit c16eafa

Please sign in to comment.