Merge pull request #248 from danskernesdigitalebibliotek/add-canary-p…

…roject

Add canary site

docs/runbooks/add-generic-site-to-platform.md

@@ -38,23 +38,19 @@ $ export DPLPLAT_ENV=dplplat01
 # instance:
 $ eval $(ssh-agent); ssh-add
 
-# 1. Authenticate against the cluster and lagoon
-$ task cluster:auth
-$ task lagoon:cli:config
-
-# 2. Add a project
+# 1. Add a project
 # PROJECT_NAME=<project name>  GIT_URL=<url> task lagoon:project:add
 $ PROJECT_NAME=dpl-cms [email protected]:danskernesdigitalebibliotek/dpl-cms.git\
   task lagoon:project:add
 
-# 2.b You can also run lagoon add project manually, consult the documentation linked
+# 1.b You can also run lagoon add project manually, consult the documentation linked
 #     in the beginning of this section for details.
 
-# 3. Deployment key
+# 2. Deployment key
 # The project is added, and a deployment key is printed. Copy it and configure
 # the GitHub repository. See the official documentation for examples.
 
-# 4. Webhook
+# 3. Webhook
 # Configure Github to post events to Lagoons webhook url.
 # The webhook url for the environment will be
 #  https://webhookhandler.lagoon.<environment>.dpl.reload.dk
@@ -64,31 +60,7 @@ $ PROJECT_NAME=dpl-cms [email protected]:danskernesdigitalebibliotek/dpl-cm
 # Referer to the official documentation linked above for an example on how to
 # set up webhooks in github.
 
-# 5. Configure image registry credentials Lagoon should use for the project
-#    IF your project references private images in repositories that requires
-#    authentication
-# Refresh your Lagoon token.
-$ lagoon login
-
-# Then export a github personal access-token with pull access.
-# We could pass this to task directly like the rest of the variables but we
-# opt for the export to keep the execution of task a bit shorter.
-$ export VARIABLE_VALUE=<github pat>
-
-# Then get the project id by listing your projects
-$ lagoon list projects
-
-# Finally, add the credentials
-$ VARIABLE_TYPE_ID=<project id> \
-  VARIABLE_TYPE=PROJECT \
-  VARIABLE_SCOPE=CONTAINER_REGISTRY \
-  VARIABLE_NAME=GITHUB_REGISTRY_CREDENTIALS \
-  task lagoon:set:environment-variable
-
-# If you get a "Invalid Auth Token" your token has probably expired, generated a
-# new with "lagoon login" and try again.
-
-# 5. Trigger a deployment manually, this will fail as the repository is empty
+# 4. Trigger a deployment manually, this will fail as the repository is empty
 #    but will serve to prepare Lagoon for future deployments.
 # lagoon deploy branch -p <project-name> -b <branch>
 $ lagoon deploy branch -p dpl-cms -b main

docs/runbooks/add-library-site-to-platform.md

@@ -66,12 +66,33 @@ sites:
 Be aware that the referenced images needs to be publicly available as Lagoon
 currently only authenticates against ghcr.io.
 
+Sites on the `webmaster` plan must have this specified as well, as this
+indicates that an environment for testing custom Drupal modules should be
+made available for the site. For example:
+
+```yaml
+sites:
+  bib-rb:
+    name: "Roskilde Bibliotek"
+    description: "Roskilde Bibliotek"
+    primary-domain: "www.roskildebib.dk"
+    secondary-domains: ["roskildebib.dk"]
+    dpl-cms-release: "1.2.3"
+    plan: webmaster
+    << : *default-release-image-source
+```
+
+The field `plan` defaults to `standard`.
+
 Then continue to provision the a Github repository for the site.
 
 ### Step 2: Provision a Github repository
 
 Run `task env_repos:provision` to create the repository.
 
+For sites with `plan: webmaster` this also creates a branch `moduletest` which
+represents the environment for testing custom Drupal modules.
+
 #### Create a Lagoon project and connect the GitHub repository
 
 Prerequisites:
@@ -92,48 +113,26 @@ The following describes a semi-automated version of "Add a Project" in
 # instance:
 $ eval $(ssh-agent); ssh-add
 
-# 1. Authenticate against the cluster and lagoon
-$ task cluster:auth
-$ task lagoon:cli:config
-
-# 2. Add a project
+# 1. Add a project
 # PROJECT_NAME=<project name>  GIT_URL=<url> task lagoon:project:add
 $ PROJECT_NAME=core-test1 [email protected]:danishpubliclibraries/env-core-test1.git\
   task lagoon:project:add
 
 # The project is added, and a deployment key is printed, use it for the next step.
 
-# 3. Add the deployment key to sites.yaml under the key "deploy_key".
+# 2. Add the deployment key to sites.yaml under the key "deploy_key".
 $ vi environments/${DPLPLAT_ENV}/sites.yaml
 # Then update the repositories using Terraform
 $ task env_repos:provision
 
-# 4. Configure image registry credentials Lagoon should use for the project:
-# Refresh your Lagoon token.
-$ lagoon login
-
-# Then export a github personal access-token with pull access.
-# We could pass this to task directly like the rest of the variables but we
-# opt for the export to keep the execution of task a bit shorter.
-$ export VARIABLE_VALUE=<github pat>
-
-# Then get the project id by listing your projects
-$ lagoon list projects
-
-# Finally, add the credentials
-$ VARIABLE_TYPE_ID=<project id> \
-  VARIABLE_TYPE=PROJECT \
-  VARIABLE_SCOPE=CONTAINER_REGISTRY \
-  VARIABLE_NAME=GITHUB_REGISTRY_CREDENTIALS \
-  task lagoon:set:environment-variable
-
-# If you get a "Invalid Auth Token" your token has probably expired, generated a
-# new with "lagoon login" and try again.
-
-# 5. Trigger a deployment manually, this will fail as the repository is empty
+# 3.a Trigger a deployment manually, this will fail as the repository is empty
 #    but will serve to prepare Lagoon for future deployments.
 # lagoon deploy branch -p <project-name> -b <branch>
 $ lagoon deploy branch -p core-test1 -b main
+
+# 3.b If you are setting up a site with `plan: webmaster`, you also need to
+# deploy the moduletest branch
+$ lagoon deploy branch -p core-test1 -b moduletest
 ```
 
 If you want to deploy a release to the site, continue to

docs/runbooks/deploy-a-release.md

@@ -8,6 +8,9 @@ or a fork to a single site.
 If you want to deploy to more than one site, simply repeat the procedure for each
 site.
 
+For sites with `plan: webmaster`, the branch `moduletest` for the test
+environment for custom Drupal modules will also be synced appropriately.
+
 ## Prerequisites
 
 * A [dplsh session](using-dplsh.md) with DPLPLAT_ENV exported and ssh-agent configured.

infrastructure/Taskfile.yml

@@ -498,6 +498,37 @@ tasks:
       - sh: "[ ! -z {{.VARIABLE_TYPE}} ]"
         msg: "Missing VARIABLE_TYPE"
 
+    lagoon:project:set:github-registry-credentials:
+      vars:
+        # VARIABLE_TYPE_ID is the PROJECT_ID. If we are given a PROJECT_NAME, lookup the ID
+        VARIABLE_TYPE_ID:
+          sh:
+            if [ ! -z {{.PROJECT_ID}} ]; then
+              echo "{{.PROJECT_ID}}";
+            else
+              lagoon get project --project "{{.PROJECT_NAME}}" --output-json | jq '.data[0].id' --raw-output;
+            fi
+        GITHUB_TOKEN:
+          sh: az keyvault secret show
+              --subscription "{{.AZURE_SUBSCRIPTION_ID}}"
+              --name github-infra-admin-pat
+              --vault-name
+                $(
+                  terraform -chdir={{.dir_infra}} output -json | jq --raw-output ".keyvault_name.value | select (.!=null)"
+                )
+              --query value -o tsv
+      cmds:
+      - task: lagoon:set:environment-variable
+        vars:
+          VARIABLE_TYPE: "PROJECT"
+          VARIABLE_SCOPE: "CONTAINER_REGISTRY"
+          VARIABLE_NAME: "GITHUB_REGISTRY_CREDENTIALS"
+          VARIABLE_TYPE_ID: "{{.VARIABLE_TYPE_ID}}"
+          VARIABLE_VALUE: "{{.GITHUB_TOKEN}}"
+      preconditions:
+      - sh: "[ ! -z {{.PROJECT_ID}} ] || [ ! -z {{.PROJECT_NAME}} ]"
+        msg: "Missing PROJECT_ID or PROJECT_NAME - at least one must be set"
+
     lagoon:add:cluster:
       deps: [cluster:auth]
       desc: Add a Kubernetes cluster (Lagoon Remote) to the Lagoon Core.
@@ -571,8 +602,9 @@ tasks:
             --openshift 1 \
             --productionEnvironment main \
             --developmentEnvironmentsLimit 25 \
-            --branches "^(main|develop)$" \
+            --branches "^(main|develop|moduletest)$" \
             --project {{.PROJECT_NAME}}
+        - task: lagoon:project:set:github-registry-credentials
         - task: lagoon:project:deploykey
       preconditions:
       - sh: "[ ! -z {{.GIT_URL}} ]"
@@ -684,6 +716,15 @@ tasks:
       env:
         SITES_CONFIG: "{{.dir_env}}/sites.yaml"
         SITE: "{{.SITE}}"
+        GITHUB_TOKEN:
+          sh: az keyvault secret show
+              --subscription "{{.AZURE_SUBSCRIPTION_ID}}"
+              --name github-infra-admin-pat
+              --vault-name
+                $(
+                  terraform -chdir={{.dir_infra}} output -json | jq --raw-output ".keyvault_name.value | select (.!=null)"
+                )
+              --query value -o tsv
       cmds:
         - dpladm/bin/sync-site.sh
       preconditions:

infrastructure/dpladm/bin/dpladm-shared.source

@@ -164,7 +164,8 @@ function syncEnvRepo {
   # Fetch a clone of the environment repo.
   local repoPath="${ENVIRONMENT_REPO_ORG}/${repoName}"
   echo "** Cloning branch ${branchName} of env-repo ${repoPath} for ${siteName}"
-  git clone --depth 1 "[email protected]:${repoPath}.git" -b "${branchName}"
+  # TODO: consider adding --single-branch flag to make cloning faster
+  git clone --depth 1 "https://${GITHUB_TOKEN}@github.com/${repoPath}.git" -b "${branchName}"
 
   # Clear out any previous content and replace it with a unrendered template.
   mv "${repoName}/.git" ./

infrastructure/dpladm/bin/sync-site.sh

@@ -77,6 +77,18 @@ function getSiteReleaseImageName {
     return
 }
 
+function getSitePlan {
+    local plan
+    plan=$(yq eval ".sites.${1}.plan" "${2}")
+    if [[ "${plan}" == "null" ]]; then
+        echo "standard"
+        return
+    fi
+
+    echo "${plan}"
+    return
+}
+
 function getSitePrimaryDomain {
     local domain
     domain=$(yq eval ".sites.${1}.primary-domain" "${2}")
@@ -135,8 +147,12 @@ siteImageRepository=$(getSiteReleaseImageRepository "${SITE}" "${SITES_CONFIG}"
 failOnErr $? "${siteImageRepository}"
 siteReleaseImageName=$(getSiteReleaseImageName "${SITE}" "${SITES_CONFIG}")
 failOnErr $? "${siteReleaseImageName}"
+plan=$(getSitePlan "${SITE}" "${SITES_CONFIG}")
 set -o errexit
 
 # Synchronise the sites environment repository.
 syncEnvRepo "${SITE}" "${releaseTag}" "${BRANCH}" "${siteImageRepository}" "${siteReleaseImageName}" "${primaryDomain}" "${secondaryDomains}"
 
+if [ "${plan}" = "webmaster" ] && [ "${BRANCH}" = "main" ]; then
+    syncEnvRepo "${SITE}" "${releaseTag}" "moduletest" "${siteImageRepository}" "${siteReleaseImageName}"
+fi

infrastructure/environments/dplplat01/sites.yaml

@@ -8,6 +8,13 @@ x-defaults:
 sites:
   # Site objects are indexed by a unique key that must be a valid lagoon, and
   # github project name. That is, alphanumeric and dashes.
+  canary:
+    name: "Canary"
+    description: "A site to test new relases on"
+    releaseImageRepository: ghcr.io/danskernesdigitalebibliotek
+    releaseImageName: dpl-cms-source
+    dpl-cms-release: "2024.10.2"
+    deploy_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFcUB4bJlX+B7upLuiOzT/3eju6l8zvor6VVmPz4n8Hp"
   kobenhavn:
     name: "Københavns Biblioteker"
     description: "The main library site for København"
@@ -43,3 +50,9 @@ sites:
     description: "The library site for Faxe"
     deploy_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBkdxUoBx0ZAXfMfA0rRUNo2EcUK39fp0M/zKJPOcYx2"
     << : *default-release-image-source
+  customizable-canary:
+    name: "Customizable bibliotek - eksempel"
+    description: "Eksempel på bibliotek der kører på 'webmaster' plan, og derfor har et modultest-miljø"
+    deploy_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILj8lXv7C/7s7te9sEpwcHQhgDWfzsCkAN7rqQ4sdTzk"
+    plan: webmaster
+    << : *default-release-image-source

infrastructure/terraform/modules/dpl-platform-env-repos/env_repositories.tf

@@ -10,6 +10,13 @@ resource "github_repository" "site" {
   archive_on_destroy = false
 }
 
+resource "github_branch" "moduletest_branch" {
+  for_each = { for key, val in local.sites : key => val if try(val.plan, "standard") == "webmaster" }
+
+  branch     = "moduletest"
+  repository = github_repository.site[each.key].name
+}
+
 # Grant the default teams their respective permissions on the repository.
 resource "github_team_repository" "site_team_default_read" {
   for_each = local.sites