Promotion 2024-12-29 anvilprod (#6789)

.github/ISSUE_TEMPLATE/upgrade.md

@@ -39,6 +39,7 @@ _period: 14 days
     - [ ] … update to [GitLab](https://hub.docker.com/r/gitlab/gitlab-ce/tags) & [GitLab runner images](https://hub.docker.com/r/gitlab/gitlab-runner/tags) <sub>or no update available</sub>
     - [ ] … update to [ClamAV image](https://hub.docker.com/r/clamav/clamav/tags) <sub>or no update available</sub>
     - [ ] … update to [GitLab AMI](https://github.com/DataBiosphere/azul/blob/develop/OPERATOR.rst#updating-the-ami-for-gitlab-instances) <sub>or no update available</sub>
+    - [ ] … update to [Swagger UI](https://github.com/DataBiosphere/azul/blob/develop/OPERATOR.rst#updating-the-swagger-ui) <sub>or no update available</sub>
 - [ ] Created tickets for any deferred updates to …
   - [ ] … to next major or minor Python version <sub>or such ticket already exists</sub>
   - [ ] … to next major Docker version <sub>or such ticket already exists</sub>

.github/PULL_REQUEST_TEMPLATE/anvilprod-hotfix.md

@@ -77,6 +77,8 @@ Connected issue: #0000
 - [ ] Started reindex in `anvilprod` <sub>or neither this PR nor a failed, prior promotion requires it</sub>
 - [ ] Checked for, triaged and possibly requeued messages in both fail queues in `anvilprod` <sub>or neither this PR nor a failed, prior promotion requires it</sub>
 - [ ] Emptied fail queues in `anvilprod` <sub>or neither this PR nor a failed, prior promotion requires it</sub>
+- [ ] Restarted the `trigger_child` job in the most recent Data Browser build for the [ucsc/anvil/anvilprod branch](https://gitlab.explore.anvilproject.org/ucsc/data-browser/-/pipelines?scope=branches) on GitLab in `anvilprod` <sub>or neither this PR nor a failed, prior promotion requires it</sub>
+- [ ] Restarted `deploy` job in the GitLab pipeline for this PR in `anvilprod` <sub>or neither this PR nor a failed, prior promotion requires it</sub>
 - [ ] Created backport PR and linked to it in a comment on this PR
 
 

.github/PULL_REQUEST_TEMPLATE/anvilprod-promotion.md

@@ -91,6 +91,8 @@ Connected issue: #0000
 - [ ] Started reindex in `anvilprod` <sub>or this PR does not require reindexing `anvilprod`</sub>
 - [ ] Checked for, triaged and possibly requeued messages in both fail queues in `anvilprod` <sub>or this PR does not require reindexing `anvilprod`</sub>
 - [ ] Emptied fail queues in `anvilprod` <sub>or this PR does not require reindexing `anvilprod`</sub>
+- [ ] Restarted the `trigger_child` job in the most recent Data Browser build for the [ucsc/anvil/anvilprod branch](https://gitlab.explore.anvilproject.org/ucsc/data-browser/-/pipelines?scope=branches) on GitLab in `anvilprod` <sub>or this PR does not require reindexing `anvilprod`</sub>
+- [ ] Restarted `deploy` job in the GitLab pipeline for this PR in `anvilprod` <sub>or this PR does not require reindexing `anvilprod`</sub>
 
 
 ### Operator

.github/PULL_REQUEST_TEMPLATE/prod-hotfix.md

@@ -72,6 +72,9 @@ Connected issue: #0000
 - [ ] Started reindex in `prod` <sub>or neither this PR nor a failed, prior promotion requires it</sub>
 - [ ] Checked for, triaged and possibly requeued messages in both fail queues in `prod` <sub>or neither this PR nor a failed, prior promotion requires it</sub>
 - [ ] Emptied fail queues in `prod` <sub>or neither this PR nor a failed, prior promotion requires it</sub>
+- [ ] Restarted the `trigger_child` job in the most recent Data Browser build for the [ucsc/hca/prod branch](https://gitlab.azul.data.humancellatlas.org/ucsc/data-browser/-/pipelines?scope=branches) on GitLab in `prod` <sub>or neither this PR nor a failed, prior promotion requires it</sub>
+- [ ] Restarted the `trigger_child` job in the most recent Data Browser build for the [ucsc/lungmap/prod branch](https://gitlab.azul.data.humancellatlas.org/ucsc/data-browser/-/pipelines?scope=branches) on GitLab in `prod` <sub>or neither this PR nor a failed, prior promotion requires it</sub>
+- [ ] Restarted `deploy` job in the GitLab pipeline for this PR in `prod` <sub>or neither this PR nor a failed, prior promotion requires it</sub>
 - [ ] Created backport PR and linked to it in a comment on this PR
 
 

.github/PULL_REQUEST_TEMPLATE/prod-promotion.md

@@ -86,6 +86,9 @@ Connected issue: #0000
 - [ ] Started reindex in `prod` <sub>or this PR does not require reindexing `prod`</sub>
 - [ ] Checked for, triaged and possibly requeued messages in both fail queues in `prod` <sub>or this PR does not require reindexing `prod`</sub>
 - [ ] Emptied fail queues in `prod` <sub>or this PR does not require reindexing `prod`</sub>
+- [ ] Restarted the `trigger_child` job in the most recent Data Browser build for the [ucsc/hca/prod branch](https://gitlab.azul.data.humancellatlas.org/ucsc/data-browser/-/pipelines?scope=branches) on GitLab in `prod` <sub>or this PR does not require reindexing `prod`</sub>
+- [ ] Restarted the `trigger_child` job in the most recent Data Browser build for the [ucsc/lungmap/prod branch](https://gitlab.azul.data.humancellatlas.org/ucsc/data-browser/-/pipelines?scope=branches) on GitLab in `prod` <sub>or this PR does not require reindexing `prod`</sub>
+- [ ] Restarted `deploy` job in the GitLab pipeline for this PR in `prod` <sub>or this PR does not require reindexing `prod`</sub>
 
 
 ### Operator

.github/pull_request_template.md

@@ -185,6 +185,11 @@ title is `Fix: ` followed by the issue title
 - [ ] Checked for, triaged and possibly requeued messages in both fail queues in `anvildev` <sub>or this PR does not require reindexing `anvildev`</sub>
 - [ ] Emptied fail queues in `dev` <sub>or this PR does not require reindexing `dev`</sub>
 - [ ] Emptied fail queues in `anvildev` <sub>or this PR does not require reindexing `anvildev`</sub>
+- [ ] Restarted the `trigger_child` job in the most recent Data Browser build for the [ucsc/hca/dev branch](https://gitlab.dev.singlecell.gi.ucsc.edu/ucsc/data-browser/-/pipelines?scope=branches) on GitLab in `dev` <sub>or this PR does not require reindexing `dev`</sub>
+- [ ] Restarted the `trigger_child` job in the most recent Data Browser build for the [ucsc/lungmap/dev branch](https://gitlab.dev.singlecell.gi.ucsc.edu/ucsc/data-browser/-/pipelines?scope=branches) on GitLab in `dev` <sub>or this PR does not require reindexing `dev`</sub>
+- [ ] Restarted `deploy` job in the GitLab pipeline for this PR in `dev` <sub>or this PR does not require reindexing `dev`</sub>
+- [ ] Restarted the `trigger_child` job in the most recent Data Browser build for the [ucsc/anvil/anvildev branch](https://gitlab.anvil.gi.ucsc.edu/ucsc/data-browser/-/pipelines?scope=branches) on GitLab in `anvildev` <sub>or this PR does not require reindexing `anvildev`</sub>
+- [ ] Restarted `deploy` job in the GitLab pipeline for this PR in `anvildev` <sub>or this PR does not require reindexing `anvildev`</sub>
 
 
 ### Operator

.github/pull_request_template.md.template.py

@@ -5,6 +5,7 @@
 from itertools import (
     chain,
 )
+import json
 from pathlib import (
     Path,
 )
@@ -26,6 +27,7 @@
 )
 
 from azul import (
+    cache,
     config,
     iif,
 )
@@ -212,6 +214,7 @@ def labels_to_promote(self, target_branch: str) -> AbstractSet[str]:
     def needs_shared_deploy(self):
         return self not in (T.backport, T.hotfix)
 
+    # noinspection PyUnusedLocal
     def shared_deploy_is_two_phase(self, target_branch: str) -> bool:
         # All `shared` components are deployed in two-phases. The first phase,
         # prior to the merge, mirrors new images to ECR, while the second phase
@@ -238,14 +241,20 @@ def main():
             emit(t, target_branch)
 
 
-def env_var(deployment: str, variable: str) -> str:
-    """
-    Return an environment variable value from the given deployment.
-    """
+@cache
+def deployment_env(deployment: str,
+                   component: str | None = None
+                   ) -> Mapping[str, str]:
     script = load_script('export_environment')
+    deployment += '' if component is None else '.' + component
     env, warning = script.load_env(deployment)
+    assert warning is None, warning
     resolved_env = script.resolve_env(env)
-    return resolved_env[variable]
+    return resolved_env
+
+
+def azul_domain_name(d):
+    return deployment_env(d)['AZUL_DOMAIN_NAME']
 
 
 def emit(t: T, target_branch: str):
@@ -334,7 +343,7 @@ def emit(t: T, target_branch: str):
             },
             iif(t is not T.backport, {
                 'type': 'cli',
-                'content': f"PR title references {t.issues('all', 'the')} connected {t.issues}"
+                'content': f'PR title references {t.issues('all', 'the')} connected {t.issues}'
             }),
             *(
                 [
@@ -729,7 +738,7 @@ def emit(t: T, target_branch: str):
                     {
                         'type': 'cli',
                         'content': f'Background migrations for '
-                                   f'[`{d}.gitlab`](https://gitlab.{env_var(d, "AZUL_DOMAIN_NAME")}'
+                                   f'[`{d}.gitlab`](https://gitlab.{azul_domain_name(d)}'
                                    f'/admin/background_migrations) are complete',
                         'alt': 'or this PR is not labeled `deploy:gitlab`'
                     }
@@ -956,12 +965,35 @@ def emit(t: T, target_branch: str):
                             for action in [
                                 'Started reindex',
                                 'Checked for, triaged and possibly requeued messages in both fail queues',
-                                'Emptied fail queues',
+                                'Emptied fail queues'
                             ]
                         ]
                     ]
                     for d, s in t.target_deployments(target_branch).items()
                 ))),
+                *[
+                    {
+                        'type': 'cli',
+                        'content': f'{action} in `{d}`',
+                        'alt': (
+                            'or neither this PR nor a failed, prior promotion requires it'
+                            if t is T.hotfix else
+                            f'or this PR does not require reindexing `{d}`'
+                        )
+                    }
+                    for d, s in t.target_deployments(target_branch).items()
+                    for action in [
+                        *[
+                            'Restarted the `trigger_child` job in the most recent Data Browser build for the ['
+                            f'{browser_site['branch']} branch]'
+                            f'(https://gitlab.{azul_domain_name(d)}/ucsc/data-browser/-/pipelines?scope=branches) '
+                            f'on GitLab'
+                            for browser_site in
+                            json.loads(deployment_env(d, 'browser')['azul_browser_sites']).values()
+                        ],
+                        'Restarted `deploy` job in the GitLab pipeline for this PR'
+                    ]
+                ],
                 iif(t is T.hotfix, {
                     'type': 'cli',
                     'content': 'Created backport PR and linked to it in a comment on this PR'

.github/workflows/ci.yml

@@ -3,7 +3,7 @@ name: 'CI'
 on:
   pull_request:
   push:
-    branches: [ 'develop', 'prod' ]
+    branches: [ 'develop', 'prod', 'anvilprod' ]
 
 jobs:
 

OPERATOR.rst

@@ -1,4 +1,5 @@
 .. contents::
+.. contents::
 
 Getting started as operator
 ---------------------------
@@ -458,6 +459,30 @@ SSH into the instance, and run ``sudo yum update`` followed by ``sudo reboot``.
 Wait for the GitLab web application to become available again and perform a
 ``git fetch`` from one of the Git repositories hosted on that instance.
 
+Updating the Swagger UI
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Operators should regularly check for available updates to the Swagger UI. The
+current version used by Azul is hardcoded in ``scripts/update_swagger.py``. The
+upstream source is located here:
+
+https://github.com/swagger-api/swagger-ui/tree/master/dist
+
+Scheduled upgrade PR's should only include minor and hotfix updates to the
+Swagger UI. If a new major version is available, open a new issue instead. To
+perform the update, edit the ``tag`` variable in the ``update_swagger`` script
+and run it. If there are nontrivial changes to the ``swagger-initializer.js`` or
+``oauth2-redirect.html`` files, cancel the update and open a new issue instead.
+Otherwise, forward any changes to those two files to their respective mustache
+template files, and commit the changes to the script and all modified files in
+the ``swagger/`` directory. The commit message must include the new tag, as well
+as a link to the upstream source in the commit body, e.g.::
+
+    Update Swagger UI to v<release version> (#issue-number)
+
+    https://github.com/swagger-api/swagger-ui/tree/v<release version>/dist
+
+
 Export AWS Inspector findings
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -605,7 +630,7 @@ backport PR first. The new PR will include the changes from the old one.
 Deploying the Data Browser
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-The Data Browser is deployed two steps. The first step is building the
+The Data Browser is deployed in two steps. The first step is building the
 ``ucsc/data-browser`` project on GitLab. This is initiated by pushing a branch
 whose name matches ``ucsc/*/*`` to one of our GitLab instances. The resulting
 pipeline produces a tarball stored in the package registry on that GitLab
@@ -616,14 +641,16 @@ downloads the tarball from the package registry and unpacks that tarball to the
 S3 bucket backing the Data Browser's CloudFront distribution.
 
 Typically, CC requests the deployment of a Data Browser instance on Slack,
-specifying the commit they wish to be deployed. After the system administrator
-approves that request, the operator merges the specified commit into one of the
-``ucsc/{atlas}/{deployment}`` branches and then pushes that branch to the
+specifying the commit (tag or sha1) they wish to be deployed. After the
+system administrator approves that request, the operator pushes the specified tag
+(if a tag was specified) to the GitLab instance for the Azul ``{deployment}``
+that backs the Data Browser instance to be deployed. Then the specified tag (or
+commit, if no tag was specified) is merged into one of the
+``ucsc/{atlas}/{deployment}`` branches. That branch is then is pushed to the
 ``DataBiosphere/data-browser`` project on GitHub, and the ``ucsc/data-browser``
-project on the GitLab instance for the Azul ``{deployment}`` that backs the Data
-Browser instance to be deployed. For the merge commit title, SmartGit's default
-can be used, as long as the title reflects the commit (branch, tag, or sha1)
-specified by CC.
+project on GitLab (same instance as above). For the merge commit title,
+SmartGit's default can be used, as long as the title reflects the commit (tag or
+sha1) specified by CC.
 
 The ``{atlas}`` placeholder can be ``hca``, ``anvil`` or ``lungmap``. Not all
 combinations of ``{atlas}`` and ``{deployment}`` are valid. Valid combinations

deployments/anvilbox/environment.py

@@ -68,7 +68,7 @@ def env() -> Mapping[str, Optional[str]]:
     other environment variables in the form `{FOO}` where FOO is the name of an
     environment variable. See
 
-    https://docs.python.org/3.11/library/string.html#format-string-syntax
+    https://docs.python.org/3.12/library/string.html#format-string-syntax
 
     for the concrete syntax. These references will be resolved *after* the
     overall environment has been compiled by merging all relevant
@@ -128,8 +128,7 @@ def env() -> Mapping[str, Optional[str]]:
 
         **(
             {
-                # $0.382/h × 2 × 24h/d × 30d/mo = $550.08/mo
-                'AZUL_ES_INSTANCE_TYPE': 'r6gd.xlarge.elasticsearch',
+                'AZUL_ES_INSTANCE_TYPE': 'r6gd.large.elasticsearch',
                 'AZUL_ES_INSTANCE_COUNT': '2',
             } if is_sandbox else {
                 # Personal deployments share an ES domain with `anvilbox`

deployments/anvildev.browser/environment.py

@@ -14,7 +14,7 @@ def env() -> Mapping[str, Optional[str]]:
     other environment variables in the form `{FOO}` where FOO is the name of an
     environment variable. See
 
-    https://docs.python.org/3.11/library/string.html#format-string-syntax
+    https://docs.python.org/3.12/library/string.html#format-string-syntax
 
     for the concrete syntax. These references will be resolved *after* the
     overall environment has been compiled by merging all relevant

deployments/anvildev.gitlab/environment.py

@@ -13,7 +13,7 @@ def env() -> Mapping[str, Optional[str]]:
     other environment variables in the form `{FOO}` where FOO is the name of an
     environment variable. See
 
-    https://docs.python.org/3.11/library/string.html#format-string-syntax
+    https://docs.python.org/3.12/library/string.html#format-string-syntax
 
     for the concrete syntax. These references will be resolved *after* the
     overall environment has been compiled by merging all relevant

deployments/anvildev/environment.py

@@ -66,7 +66,7 @@ def env() -> Mapping[str, Optional[str]]:
     other environment variables in the form `{FOO}` where FOO is the name of an
     environment variable. See
 
-    https://docs.python.org/3.11/library/string.html#format-string-syntax
+    https://docs.python.org/3.12/library/string.html#format-string-syntax
 
     for the concrete syntax. These references will be resolved *after* the
     overall environment has been compiled by merging all relevant
@@ -81,11 +81,11 @@ def env() -> Mapping[str, Optional[str]]:
         # Set variables for the `anvildev` (short for AnVIL development)
         # deployment here.
         #
-        # Only modify this file if you intend to commit those changes. To change the
-        # environment with a setting that's specific to you AND the deployment, create
-        # a environment.local.py right next to this file and make your changes there.
-        # Settings applicable to all environments but specific to you go into
-        # environment.local.py at the project root.
+        # Only modify this file if you intend to commit those changes. To apply
+        # a setting that's specific to you AND the deployment, create an
+        # `environment.local.py` file right next to this one and apply that
+        # setting there. Settings that are applicable to all environments but
+        # specific to you go into `environment.local.py` at the project root.
 
         'AZUL_DEPLOYMENT_STAGE': 'anvildev',
 
@@ -115,8 +115,7 @@ def env() -> Mapping[str, Optional[str]]:
 
         'AZUL_ENABLE_MONITORING': '1',
 
-        # $0.382/h × 3 × 24h/d × 30d/mo = $825.12/mo
-        'AZUL_ES_INSTANCE_TYPE': 'r6gd.xlarge.elasticsearch',
+        'AZUL_ES_INSTANCE_TYPE': 'r6gd.large.elasticsearch',
         'AZUL_ES_INSTANCE_COUNT': '2',
 
         'AZUL_DEBUG': '1',

deployments/anvilprod.browser/environment.py

@@ -14,7 +14,7 @@ def env() -> Mapping[str, Optional[str]]:
     other environment variables in the form `{FOO}` where FOO is the name of an
     environment variable. See
 
-    https://docs.python.org/3.11/library/string.html#format-string-syntax
+    https://docs.python.org/3.12/library/string.html#format-string-syntax
 
     for the concrete syntax. These references will be resolved *after* the
     overall environment has been compiled by merging all relevant

deployments/anvilprod.gitlab/environment.py

@@ -13,7 +13,7 @@ def env() -> Mapping[str, Optional[str]]:
     other environment variables in the form `{FOO}` where FOO is the name of an
     environment variable. See
 
-    https://docs.python.org/3.11/library/string.html#format-string-syntax
+    https://docs.python.org/3.12/library/string.html#format-string-syntax
 
     for the concrete syntax. These references will be resolved *after* the
     overall environment has been compiled by merging all relevant

deployments/anvilprod.shared/environment.py

@@ -13,7 +13,7 @@ def env() -> Mapping[str, Optional[str]]:
     other environment variables in the form `{FOO}` where FOO is the name of an
     environment variable. See
 
-    https://docs.python.org/3.11/library/string.html#format-string-syntax
+    https://docs.python.org/3.12/library/string.html#format-string-syntax
 
     for the concrete syntax. These references will be resolved *after* the
     overall environment has been compiled by merging all relevant