Sites.yaml: source of truth and driver of infra #246
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Base automatically changed from
staging-envs-for-customizable-plan
to
add-canary-project
March 14, 2024 10:29
hypesystem
force-pushed
the
sites.yaml-source-of-truth
branch
from
57bd5de to
961f7f1
Compare
Merged
spaceo
reviewed
Apr 8, 2024
spaceo
reviewed
Apr 8, 2024
|
|
||
| The following describes a semi-automated version of "Add a Project" in | ||
| [the official documentation](https://docs.lagoon.sh/installing-lagoon/add-project/). | ||
| From within `dplsh` run the `sites:sync` task to sync the site state in |
There was a problem hiding this comment.
So the only Prerequisity here would be that you are allowed to push (via an ssh key) to the dpl-platform repo?
There was a problem hiding this comment.
Well, it is mentioned later under "Synchronize site state for all sites" that you also need an ssh-key added via the Lagoon UI. I could maybe merge the sections now that I've rewritten much of it?
spaceo
reviewed
Apr 8, 2024
spaceo
reviewed
Apr 8, 2024
Comment on lines
+35
to
+41
| # 3a. For project-level variables, use the following command, which creates | ||
| # the variable if it does not yet exist, or updates it otherwise: | ||
| $ task lagoon:ensure:environment-variable \ | ||
| PROJECT_NAME=<project name> \ | ||
| VARIABLE_SCOPE=RUNTIME \ | ||
| VARIABLE_NAME=<your variable name> \ | ||
| VARIABLE_VALUE=<your variable value> |
There was a problem hiding this comment.
Would be nice with an example of a runtime/environment level variable that you could set
There was a problem hiding this comment.
Hmm... you can kind of set anything you want, it is just passed as an environment variable to the cli and php services.
I'm not sure if we have any good examples where we modify behavior based on env vars -- but it would be an obvious way to differentiate plans, if we need to, for example.
spaceo
reviewed
Apr 8, 2024
spaceo
reviewed
Apr 8, 2024
Comment on lines
+530
to
+545
| preconditions: | ||
| - sh: "[ ! -z {{.VARIABLE_VALUE}} ]" | ||
| msg: "Missing VARIABLE_VALUE" | ||
| - sh: "[ ! -z {{.VARIABLE_NAME}} ]" | ||
| msg: "Missing VARIABLE_NAME" | ||
| - sh: "[ ! -z {{.VARIABLE_SCOPE}} ]" | ||
| msg: "Missing VARIABLE_SCOPE" | ||
| - sh: "[ ! -z {{.PROJECT_NAME}} ]" | ||
| msg: "Missing PROJECT_NAME" |
There was a problem hiding this comment.
In the latest versions of Task you can use the requires property instead. I think it is a bit more clean:
spaceo
reviewed
Apr 8, 2024
spaceo
reviewed
Apr 8, 2024
Comment on lines
+830
to
+839
| preconditions: | ||
| - sh: "[ ! -z {{.SITE}} ]" | ||
| msg: "Env variable SITE is not set or empty." |
There was a problem hiding this comment.
You could use requires here (and in multiple other tasks)....
spaceo
reviewed
Apr 8, 2024
spaceo
reviewed
Apr 8, 2024
hypesystem
force-pushed
the
sites.yaml-source-of-truth
branch
from
f278242 to
0ac2aba
Compare
|
In the name of getting things out I'll push these to later PRs:
|
The new task `sites:sync` syncs *all* sites from sites.yaml, ensuring that everything is set up for them as it should. This includes setting up Lagoon with GITHUB token, capturing the Lagoon deploy key, saving it in sites.yaml, and ensuring all the Github repos are provisioned with Terraform. A single case is missing right now, which is running a first `lagoon deploy` for new sites. We need to figure out if that is actually required, and if so, if we can detect whether it needs doing
…e on environment variables This avoids the issue of setting github credentials for already existing sites failing, because we were using a create-only method. So now we update setting github token to use this new method and can generalize when we set the token
The runbook is superceded by the new runbook for `add-library-site-to-platform`.
…ting method When setting environment variables in Lagoon the new method has these advantages: - Does not fail when variable already exists, so learning to use this method means it will be easier to write Taskfile tasks with this knowledge - Does not require looking up IDs before sending the command - it takes the project and environment names we already use to refer to these entities, and so provides a more natural interface.
hypesystem
force-pushed
the
sites.yaml-source-of-truth
branch
from
dbdb06b to
c3cc0f8
Compare
yq defaults to adding "!!merge" as a prefix to the now non-standard `<<:` key. This left terraform yamldecode unable to parse the file. The added code ensures we do not set the "!!merge" prefix.
They are a known issue and will be fixed soon. Right now the errors have no effect on the correctness of the deployment
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Simplifies infrastructure alignment with sites.yaml significantly by replacing several runbooks with a single
sites:synctask, which ensures that all sites are in correct state as described in sites.yaml.This means less work when creating new sites, updating information, or changing templating for content in sites in the future. We give sites.yaml more power and require less manual work.
In the future we can then expand on the sites.yaml spec to drive what we need to happen in the infrastructure, in line with the principles in ADR 004: Declarative Site management but taking the principle slightly further and aligning on a single source of truth. This is covered in the new ADR 005 which is added in this PR.
Should this be tested by the reviewer and how?
For now - await my green light on actually running the code.
But please review the code and documentation changes for clarity.
Later: (DO NOT DO THIS NOW)
task sites:syncand see that it works.What are the relevant tickets?
DDRIFT-45