Merge pull request #246 from danskernesdigitalebibliotek/sites.yaml-s…

…ource-of-truth

Sites.yaml: source of truth and driver of infra

docs/architecture/adr/adr-005-declarative-site-management-2.md

@@ -0,0 +1,48 @@
+# ADR 005: Declarative Site management (2)
+
+## Context
+
+We have previously established ([in ADR 004](adr-004-declarative-site-management.md))
+that we want to take a declarative approach to site management. The previous
+ADR focuses on using a single declarative file, `sites.yaml`, for driving
+Github repositories that can be used by Lagoon to run sites.
+
+The considerations from that ADR apply equally here. We have identified more
+opportunities to use a declarative approach, which will simultaneously
+significantly simplify the work required for platform maintainers when managing
+sites.
+
+Specifically, runbooks with several steps to effectuate a new deployment are a
+likely source of errors. The same can be said of having to manually run
+commands to make changes in the platform.
+
+Every time we run a command that is not documented in the source code in the
+platform `main` branch, it becomes less clear what the state of the platform is.
+
+Conversely, every time a change is made in the `main` branch that has not yet
+been executed, it becomes less clear what the state of the platform is.
+
+## Decision
+
+We continuously strive towards making the `main` branch in the dpl-platform repo
+the single source of truth for what the state of the platform should be. The
+repository becomes the declaration of the entire state.
+
+This leads to at least two concrete steps:
+
+- We will automate synchronizing state in the platform-repo with the actual
+  platform state. This means using `sites.yaml` to declare the expected state
+  in Lagoon (e.g. which projects and environments are created), not just the
+  Github repos. This leaves the deployment process less error prone.
+
+- We will automate running the synchronization step every time code is checked
+  into the `main` branch. This means state divergence between the platform repo
+  declaration and reality is minimized.
+
+It will *still* be possible for `dpl-cms` to maintain its own area of state
+in the platform, but anything declared in `dpl-platform` will be much more
+likely to be the actual state in the platform.
+
+## Status
+
+Proposed

docs/architecture/platform-environment-architecture.md

@@ -136,8 +136,7 @@ file shared by all sites on the platform.
 
 Consult the following runbooks to see the procedures for:
 
-* [Adding a site to the platform](../runbooks/add-site-to-platform.md)
-* [Deploying to a site](../runbooks/deploy-a-release.md)
+* [Adding a site to the platform](../runbooks/add-library-site-to-platform.md)
 * [Removing a site](../runbooks/remove-site-from-platform.md)
 
 ### sites.yaml

docs/runbooks/add-library-site-to-platform.md

@@ -17,18 +17,17 @@ site to the platform.
 The following sections describes how to
 
 * Add the site to `sites.yaml`
-* Provision Github "environment" repository
-* Create a Lagoon project and connect it to the repository
-
-After these steps has been completed, you can continue to deploying to the
-site. See the [deploy-a-release.md](deploy-a-release.md) for details.
+* Using the `sites.yaml` specification to provision Github repositories,
+  create Lagoon projects, tie the two together, and deploy all the
+  relevant environments.
 
 ### Step 1, update sites.yaml
 
 Create an entry for the site in `sites.yaml`.
 
-For now specify an unique site key (its key in the map of sites), name and
-description. Leave out the deployment-key, you will add it in a later step.
+Specify a unique site key (its key in the map of sites), name and description.
+Leave out the deployment-key, it will be added automatically by the
+synchronization script.
 
 Sample entry (beware that this example be out of sync with the environment you
 are operating, so make sure to compare it with existing entries from the
@@ -66,9 +65,9 @@ sites:
 Be aware that the referenced images needs to be publicly available as Lagoon
 currently only authenticates against ghcr.io.
 
-Sites on the `webmaster` plan must have this specified as well, as this
-indicates that an environment for testing custom Drupal modules should be
-made available for the site. For example:
+Sites on the "webmaster" plan must have the field `plan` set to `"webmaster"`
+as this indicates that an environment for testing custom Drupal modules should
+also be deployed. For example:
 
 ```yaml
 sites:
@@ -84,59 +83,43 @@ sites:
 
 The field `plan` defaults to `standard`.
 
-Then continue to provision the a Github repository for the site.
-
-### Step 2: Provision a Github repository
+Now you are ready to sync the site state.
 
-Run `task env_repos:provision` to create the repository.
+### Synchronize site state for all sites
 
-For sites with `plan: webmaster` this also creates a branch `moduletest` which
-represents the environment for testing custom Drupal modules.
+You have made a single additive change to the `sites.yaml` file. It is
+important to ensure that your branch is otherwise up-to-date with `main`,
+as the state you currently have in `sites.yaml` is what will be ensured exists
+in the platform.
 
-#### Create a Lagoon project and connect the GitHub repository
+You may end up undoing changes that other people have done, if you don't have
+their changes to `sites.yaml` in your branch.
 
 Prerequisites:
 
 * A Lagoon account on the Lagoon core with your ssh-key associated (created through
   the Lagoon UI, on the Settings page)
-* The git-url for the sites environment repository (you don't need to create this
-  repository - it will be created by the task below - but the URL must match the
-  Github repository that will be created)
-* A personal access-token that is allowed to pull images from the image-registry
-  that hosts our images.
-* The platform environment name (Consult the [platform environment documentation](https://github.com/danskernesdigitalebibliotek/dpl-platform/wiki/Platform-Environments))
 
-The following describes a semi-automated version of "Add a Project" in
-[the official documentation](https://docs.lagoon.sh/installing-lagoon/add-project/).
+From within `dplsh` run the `sites:sync` task to sync the site state in
+sites.yaml, creating your new site
 
 ```sh
-# From within dplsh:
-
-# If your ssh-key is passphrase-projected we'll need to setup an ssh-agent
-# instance:
-$ eval $(ssh-agent); ssh-add
-
-# 1. Add a project
-# PROJECT_NAME=<project name>  GIT_URL=<url> task lagoon:project:add
-$ PROJECT_NAME=core-test1 [email protected]:danishpubliclibraries/env-core-test1.git\
-  task lagoon:project:add
-
-# The project is added, and a deployment key is printed, use it for the next step.
+task sites:sync
+```
 
-# 2. Add the deployment key to sites.yaml under the key "deploy_key".
-$ vi environments/${DPLPLAT_ENV}/sites.yaml
-# Then update the repositories using Terraform
-$ task env_repos:provision
+You may be prompted to confirm Terraform plan execution and approve other
+critical steps. Read and consider these messages carefully and ensure you are
+not needlessly changing other sites.
 
-# 3.a Trigger a deployment manually, this will fail as the repository is empty
-#    but will serve to prepare Lagoon for future deployments.
-# lagoon deploy branch -p <project-name> -b <branch>
-$ lagoon deploy branch -p core-test1 -b main
+The synchronization process:
 
-# 3.b If you are setting up a site with `plan: webmaster`, you also need to
-# deploy the moduletest branch
-$ lagoon deploy branch -p core-test1 -b moduletest
-```
+* ensures a Github repo is provisioned for each site
+* creates a Lagoon configuration for each site and pushes it to the relevant
+  branches in the repo (for example, sites with `plan: "webmaster"` also get
+  a `moduletest` branch for testing custom Drupal modules)
+* ensures a Lagoon project is created for each site
+* configures Lagoon to track and deploy all the relevant branches for each site
+  as environments
 
-If you want to deploy a release to the site, continue to
-[Deploying a release](deploy-a-release.md).
+If no other changes have been made to `sites.yaml`, the result is that your new
+site is created and deployed to all relevant environments.

docs/runbooks/set-environment-variable.md

@@ -17,7 +17,8 @@ picked up and parsed eg. in Drupals `settings.php`.
 * A running [dplsh](using-dplsh.md) with `DPLPLAT_ENV` set to the platform
   environment name and ssh-agent running if your ssh-keys are passphrase
   protected.
-* An user that has administrative privileges on the lagoon project in question.
+* A Lagoon account on the Lagoon core with your ssh-key associated (created
+  through the Lagoon UI, on the Settings page)
 
 ## Procedure
 
@@ -31,29 +32,22 @@ $ task lagoon:cli:config
 # 2. Refresh your Lagoon token.
 $ lagoon login
 
-# 3. Then get the project id by listing your projects
-$ lagoon list projects
-
-# 4. Optionally, if you wish to set a variable for a single environment - eg a
-# pull-request site, list the environments in the project
-$ lagoon list environments -p <project name>
+# 3a. For project-level variables, use the following command, which creates
+#     the variable if it does not yet exist, or updates it otherwise:
+$ task lagoon:ensure:environment-variable \
+  PROJECT_NAME=<project name> \
+  VARIABLE_SCOPE=RUNTIME \
+  VARIABLE_NAME=<your variable name> \
+  VARIABLE_VALUE=<your variable value>
 
-# 5. Finally, set the variable
-# - Set the the type_id to the id of the project or environment depending on
-#   whether you want all or a single environment to access the variable
-# - Set scope to VARIABLE_TYPE or ENVIRONMENT depending on the choice above
-# - set
-$ VARIABLE_TYPE_ID=<project or environment id> \
-  VARIABLE_TYPE=<PROJECT or ENVIRONMENT> \
+# 3b. Or, to similarly ensure the value of an environment-level variable:
+$ task lagoon:ensure:environment-variable \
+  PROJECT_NAME=<project name> \
+  ENVIRONMENT_NAME=<environment name> \
   VARIABLE_SCOPE=RUNTIME \
   VARIABLE_NAME=<your variable name> \
-  VARIABLE_VALUE=<your variable value> \
-  task lagoon:set:environment-variable
+  VARIABLE_VALUE=<your variable value>
 
 # If you get a "Invalid Auth Token" your token has probably expired, generated a
 # new with "lagoon login" and try again.
 ```
-
-The variable will be available the next time the site is deployed by Lagoon.
-Use [the deployment runbook](deploy-a-release.md) to trigger a deployment, use
-a forced deployment if you do not have a new release to deploy.