[R] Upgrade direct Python dependencies (#6511, PR #6649)

DataBiosphere · Nov 26, 2024 · 7ea685b · 7ea685b
2 parents f08b173 + e3686fc
commit 7ea685b
Show file tree

Hide file tree

Showing 58 changed files with 234 additions and 202 deletions.
diff --git a/.github/ISSUE_TEMPLATE/upgrade.md b/.github/ISSUE_TEMPLATE/upgrade.md
@@ -28,16 +28,17 @@ _period: 14 days
   - [ ] GH Action workflow succeeded
   - [ ] Image is available on [DockerHub](https://hub.docker.com/repository/docker/ucscgi/azul-bigquery-emulator) 
 - [ ] Create Azul PR, connected to this issue, with … 
-  - [ ] … changes to `requirements*.txt` from open Dependabot PRs, one commit per PR
-  - [ ] … update to [Python](https://hub.docker.com/_/python/tags) (only patch versions) <sub>or no update available</sub>
-  - [ ] … update to [Terraform](https://hub.docker.com/r/hashicorp/terraform/tags) (only patch versions) <sub>or no update available</sub>
-  - [ ] … new [PyCharm image](https://hub.docker.com/repository/docker/ucscgi/azul-pycharm)
-  - [ ] … new [Elasticsearch image](https://hub.docker.com/repository/docker/ucscgi/azul-elasticsearch)
-  - [ ] … new [BigQuery Emulator image](https://hub.docker.com/repository/docker/ucscgi/azul-bigquery-emulator)
-  - [ ] … update to [Docker images](https://hub.docker.com/_/docker/tags) (only minor and patch versions) <sub>or no update available</sub>
-  - [ ] … update to [GitLab](https://hub.docker.com/r/gitlab/gitlab-ce/tags) & [GitLab runner images](https://hub.docker.com/r/gitlab/gitlab-runner/tags) <sub>or no update available</sub>
-  - [ ] … update to [ClamAV image](https://hub.docker.com/r/clamav/clamav/tags) <sub>or no update available</sub>
-  - [ ] … update to [GitLab AMI](https://github.com/DataBiosphere/azul/blob/develop/OPERATOR.rst#updating-the-ami-for-gitlab-instances) <sub>or no update available</sub>
+    - [ ] … changes to `requirements*.txt` from open Dependabot PRs, one commit per PR
+    - [ ] … upgrade direct Python dependencies, [reference the operator manual](https://github.com/DataBiosphere/azul/blob/develop/OPERATOR.rst#upgrade-direct-python-dependencies) for instructions <sub>or not applicable</sub>
+    - [ ] … update to [Python](https://hub.docker.com/_/python/tags) (only patch versions) <sub>or no update available</sub>
+    - [ ] … update to [Terraform](https://hub.docker.com/r/hashicorp/terraform/tags) (only patch versions) <sub>or no update available</sub>
+    - [ ] … new [PyCharm image](https://hub.docker.com/repository/docker/ucscgi/azul-pycharm)
+    - [ ] … new [Elasticsearch image](https://hub.docker.com/repository/docker/ucscgi/azul-elasticsearch)
+    - [ ] … new [BigQuery Emulator image](https://hub.docker.com/repository/docker/ucscgi/azul-bigquery-emulator)
+    - [ ] … update to [Docker images](https://hub.docker.com/_/docker/tags) (only minor and patch versions) <sub>or no update available</sub>
+    - [ ] … update to [GitLab](https://hub.docker.com/r/gitlab/gitlab-ce/tags) & [GitLab runner images](https://hub.docker.com/r/gitlab/gitlab-runner/tags) <sub>or no update available</sub>
+    - [ ] … update to [ClamAV image](https://hub.docker.com/r/clamav/clamav/tags) <sub>or no update available</sub>
+    - [ ] … update to [GitLab AMI](https://github.com/DataBiosphere/azul/blob/develop/OPERATOR.rst#updating-the-ami-for-gitlab-instances) <sub>or no update available</sub>
 - [ ] Created tickets for any deferred updates to …
   - [ ] … to next major or minor Python version <sub>or such ticket already exists</sub>
   - [ ] … to next major Docker version <sub>or such ticket already exists</sub>

diff --git a/OPERATOR.rst b/OPERATOR.rst
@@ -346,6 +346,52 @@ a step on that path per biweekly upgrade PR.
 Before upgrading the GitLab version, create a backup of the GitLab volume. See
 `Backup GitLab volumes`_ for help.
 
+Upgrade direct Python dependencies
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In PyCharm, use `Package tool window`_  to view the most recent versions of
+the project's direct Python dependencies. This feature may only work properly
+after running ``make envhook``, and correctly configuring the Python interpreter
+for the project (at least once before).
+
+Proceed by identifying the packages that are candidates for upgrades. Check the
+dependencies listed in ``requirements.txt`` and ``requirements.dev.txt`` against
+the Package tool window, where the dependency indicates of an available version.
+When updating:
+
+ * Update to the latest mature release (a release with a high patch number or
+   where the most recent patch release is at least a couple of months old) and go
+   backward if problems occur.
+
+  * Document each of these problems with a dedicated FIXME, with its respective
+    ticket & reference, when non-trivial code base changes are necessary due to
+    a package version upgrade.
+
+  * Reference the GitHub link in a comment beside the conflictive package.
+
+ * If updating a package causes a trivial change or a dismissable warning when
+   including a FIXME (e.g., deprecation warnings), it should be done on its own
+   commit, to easily identify the dependencies forcing the change and the given
+   resolution.
+
+Note, a way to display all available versions of a given package in a concise
+way, is to pretend to install a non-existing version from a terminal console
+via the pip command. For example, to see all available versions of ``flake8``
+one may run ``pip install flake8=9.9.9``, and the output will display all
+versions of the dependency.
+
+As always, each of the committed changes should be tested, and should
+independently succeed all feature branch checks in GitHub, etc. Perform the
+following for smoke-testing basic operations and functions:
+
+ #. Recreate the project's virtualenv from scratch, run the ``requirements``
+    target, run the ``envhook`` target and end with ``requirements_update``.
+
+ #. Run the ``test``, and ``deploy`` targets in personal deployment (or via
+    sandbox) and then run the integration test.
+
+.. _Package tool window: https://www.jetbrains.com/guide/python/tutorials/getting-started-pycharm/installing-and-managing-python-packages/
+
 Increase GitLab data volume size
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

diff --git a/bin/wheels/runtime/Deprecated-1.2.13-py2.py3-none-any.whl b/bin/wheels/runtime/Deprecated-1.2.13-py2.py3-none-any.whl
diff --git a/bin/wheels/runtime/Deprecated-1.2.14-py2.py3-none-any.whl b/bin/wheels/runtime/Deprecated-1.2.14-py2.py3-none-any.whl
diff --git a/bin/wheels/runtime/attrs-23.1.0-py3-none-any.whl b/bin/wheels/runtime/attrs-23.1.0-py3-none-any.whl
diff --git a/bin/wheels/runtime/attrs-24.2.0-py3-none-any.whl b/bin/wheels/runtime/attrs-24.2.0-py3-none-any.whl
diff --git a/bin/wheels/runtime/bdbag-1.6.3-py2.py3-none-any.whl b/bin/wheels/runtime/bdbag-1.6.3-py2.py3-none-any.whl
diff --git a/bin/wheels/runtime/bdbag-1.7.3-py2.py3-none-any.whl b/bin/wheels/runtime/bdbag-1.7.3-py2.py3-none-any.whl
diff --git a/...ls/runtime/boto3-1.28.63-py3-none-any.whl → ...ls/runtime/boto3-1.35.48-py3-none-any.whl b/...ls/runtime/boto3-1.28.63-py3-none-any.whl → ...ls/runtime/boto3-1.35.48-py3-none-any.whl
diff --git a/...runtime/botocore-1.31.63-py3-none-any.whl → ...runtime/botocore-1.35.48-py3-none-any.whl b/...runtime/botocore-1.31.63-py3-none-any.whl → ...runtime/botocore-1.35.48-py3-none-any.whl
diff --git a/...elasticsearch-7.17.9-py2.py3-none-any.whl → ...lasticsearch-7.17.12-py2.py3-none-any.whl b/...elasticsearch-7.17.9-py2.py3-none-any.whl → ...lasticsearch-7.17.12-py2.py3-none-any.whl
diff --git a/bin/wheels/runtime/elasticsearch_dsl-7.4.0-py2.py3-none-any.whl b/bin/wheels/runtime/elasticsearch_dsl-7.4.0-py2.py3-none-any.whl
diff --git a/bin/wheels/runtime/elasticsearch_dsl-7.4.1-py2.py3-none-any.whl b/bin/wheels/runtime/elasticsearch_dsl-7.4.1-py2.py3-none-any.whl
diff --git a/bin/wheels/runtime/fastavro-1.8.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl b/bin/wheels/runtime/fastavro-1.8.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
diff --git a/bin/wheels/runtime/fastavro-1.9.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl b/bin/wheels/runtime/fastavro-1.9.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
diff --git a/bin/wheels/runtime/google_api_core-2.12.0-py3-none-any.whl b/bin/wheels/runtime/google_api_core-2.12.0-py3-none-any.whl
diff --git a/bin/wheels/runtime/google_api_core-2.21.0-py3-none-any.whl b/bin/wheels/runtime/google_api_core-2.21.0-py3-none-any.whl
diff --git a/bin/wheels/runtime/google_auth-2.23.3-py2.py3-none-any.whl b/bin/wheels/runtime/google_auth-2.23.3-py2.py3-none-any.whl
diff --git a/bin/wheels/runtime/google_auth-2.34.0-py2.py3-none-any.whl b/bin/wheels/runtime/google_auth-2.34.0-py2.py3-none-any.whl
diff --git a/bin/wheels/runtime/google_cloud_bigquery-3.12.0-py2.py3-none-any.whl b/bin/wheels/runtime/google_cloud_bigquery-3.12.0-py2.py3-none-any.whl
diff --git a/bin/wheels/runtime/google_cloud_bigquery-3.13.0-py2.py3-none-any.whl b/bin/wheels/runtime/google_cloud_bigquery-3.13.0-py2.py3-none-any.whl
diff --git a/bin/wheels/runtime/grpcio-1.67.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl b/bin/wheels/runtime/grpcio-1.67.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
diff --git a/bin/wheels/runtime/grpcio-1.68.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl b/bin/wheels/runtime/grpcio-1.68.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
diff --git a/bin/wheels/runtime/http_message_signatures-0.4.4-py3-none-any.whl b/bin/wheels/runtime/http_message_signatures-0.4.4-py3-none-any.whl
diff --git a/bin/wheels/runtime/http_message_signatures-0.5.0-py3-none-any.whl b/bin/wheels/runtime/http_message_signatures-0.5.0-py3-none-any.whl
diff --git a/bin/wheels/runtime/more_itertools-10.5.0-py3-none-any.whl b/bin/wheels/runtime/more_itertools-10.5.0-py3-none-any.whl
diff --git a/bin/wheels/runtime/more_itertools-9.0.0-py3-none-any.whl b/bin/wheels/runtime/more_itertools-9.0.0-py3-none-any.whl
diff --git a/bin/wheels/runtime/msgpack-1.0.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl b/bin/wheels/runtime/msgpack-1.0.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
diff --git a/bin/wheels/runtime/msgpack-1.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl b/bin/wheels/runtime/msgpack-1.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
diff --git a/.../runtime/requests-2.32.2-py3-none-any.whl → .../runtime/requests-2.32.3-py3-none-any.whl b/.../runtime/requests-2.32.2-py3-none-any.whl → .../runtime/requests-2.32.3-py3-none-any.whl
diff --git a/bin/wheels/runtime/s3transfer-0.10.4-py3-none-any.whl b/bin/wheels/runtime/s3transfer-0.10.4-py3-none-any.whl
diff --git a/bin/wheels/runtime/s3transfer-0.7.0-py3-none-any.whl b/bin/wheels/runtime/s3transfer-0.7.0-py3-none-any.whl
diff --git a/bin/wheels/runtime/tzlocal-2.1-py2.py3-none-any.whl b/bin/wheels/runtime/tzlocal-2.1-py2.py3-none-any.whl
diff --git a/bin/wheels/runtime/tzlocal-5.2-py3-none-any.whl b/bin/wheels/runtime/tzlocal-5.2-py3-none-any.whl
diff --git a/...time/urllib3-1.26.19-py2.py3-none-any.whl → ...time/urllib3-1.26.20-py2.py3-none-any.whl b/...time/urllib3-1.26.19-py2.py3-none-any.whl → ...time/urllib3-1.26.20-py2.py3-none-any.whl
diff --git a/...311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl b/...311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
diff --git a/...311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl b/...311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
diff --git a/requirements.all.txt b/requirements.all.txt
@@ -1,16 +1,16 @@
 atomicwrites==1.4.1
-attrs==23.1.0
+attrs==24.2.0
 aws-requests-auth==0.4.3
-awscli==1.29.63
+awscli==1.35.14
 bagit==1.8.1
 bagit-profile==1.3.1
-bdbag==1.6.3
+bdbag==1.7.3
 blessed==1.20.0
 blinker==1.9.0
-boto3==1.28.63
-boto3-stubs==1.28.63
-botocore==1.31.63
-botocore-stubs==1.35.61
+boto3==1.35.48
+boto3-stubs==1.35.48
+botocore==1.35.48
+botocore-stubs==1.35.69
 brotli==1.1.0
 cachetools==5.5.0
 certifi==2024.8.30
@@ -19,126 +19,123 @@ chalice==1.30.0+12
 charset-normalizer==3.4.0
 chevron==0.14.0
 click==8.1.7
-colorama==0.4.4
+colorama==0.4.6
 configargparse==1.7
-coverage==6.5.0
+coverage==7.6.4
 cryptography==43.0.3
-deprecated==1.2.13
+deprecated==1.2.14
 docker==7.1.0
 docutils==0.16
-elasticsearch==7.17.9
-elasticsearch-dsl==7.4.0
+elasticsearch==7.17.12
+elasticsearch-dsl==7.4.1
 et_xmlfile==2.0.0
-fastavro==1.8.2
-flake8==5.0.4
+fastavro==1.9.7
+flake8==7.1.1
 flask==3.0.3
-flask-basicauth==0.2.0
 flask-cors==5.0.0
+flask-login==0.6.3
 furl==2.1.3
-gevent==23.9.1
-geventhttpclient==2.3.1
+gevent==24.10.2
+geventhttpclient==2.3.3
 gitdb==4.0.11
-gitpython==3.1.41
-google-api-core==2.12.0
-google-api-python-client==2.103.0
-google-auth==2.23.3
+gitpython==3.1.43
+google-api-core==2.21.0
+google-api-python-client==2.149.0
+google-auth==2.34.0
 google-auth-httplib2==0.2.0
-google-cloud-bigquery==3.12.0
-google-cloud-bigquery-reservation==1.11.3
+google-cloud-bigquery==3.13.0
+google-cloud-bigquery-reservation==1.13.5
 google-cloud-core==2.4.1
-google-cloud-storage==2.12.0
+google-cloud-storage==2.18.2
 google-crc32c==1.6.0
 google-resumable-media==2.7.2
 googleapis-common-protos==1.66.0
 greenlet==3.1.1
-grpcio==1.67.1
+grpcio==1.68.0
 grpcio-status==1.62.3
-http-message-signatures==0.4.4
+http-message-signatures==0.5.0
 http_sfv==0.9.9
 httplib2==0.22.0
 idna==3.10
-importlib-resources==5.13.0
 inquirer==2.10.1
 itsdangerous==2.2.0
-jdcal==1.4.1
 jinja2==3.1.4
 jmespath==1.0.1
-jq==1.3.0
-jsonschema==4.17.3
-jsonschema-spec==0.1.6
-jwcrypto==1.5.6
+jq==1.8.0
+jsonschema==4.23.0
+jsonschema-path==0.3.3
+jsonschema-specifications==2023.12.1
 lazy-object-proxy==1.10.0
-locust==2.12.2
+locust==2.32.0
 markupsafe==3.0.2
 mccabe==0.7.0
-more-itertools==9.0.0
-moto==4.1.13
-msgpack==1.0.7
-mypy-boto3-dynamodb==1.28.73
-mypy-boto3-ecr==1.28.45
-mypy-boto3-iam==1.28.79
-mypy-boto3-kms==1.28.37
-mypy-boto3-lambda==1.28.83
-mypy-boto3-s3==1.28.55
-mypy-boto3-sqs==1.28.82
-mypy-boto3-stepfunctions==1.28.36
-openapi-schema-validator==0.3.4
-openapi-spec-validator==0.5.1
-openpyxl==3.0.6
+more-itertools==10.5.0
+moto==5.0.18
+msgpack==1.1.0
+mypy-boto3-dynamodb==1.35.60
+mypy-boto3-ecr==1.35.21
+mypy-boto3-iam==1.35.61
+mypy-boto3-kms==1.35.0
+mypy-boto3-lambda==1.35.68
+mypy-boto3-s3==1.35.69
+mypy-boto3-sqs==1.35.0
+mypy-boto3-stepfunctions==1.35.68
+openapi-schema-validator==0.6.2
+openapi-spec-validator==0.7.1
+openpyxl==3.1.5
 orderedmultidict==1.0.1
 packaging==24.2
 pathable==0.4.3
-pip==23.3.2
+pip==24.0
 posix_ipc==1.1.1
 proto-plus==1.25.0
 protobuf==4.25.5
 psutil==6.1.0
-py-partiql-parser==0.3.3
+py-partiql-parser==0.5.6
 pyasn1==0.6.1
 pyasn1_modules==0.4.1
-pycodestyle==2.9.1
+pycodestyle==2.12.1
 pycparser==2.22
-pyflakes==2.5.0
-pygithub==1.56
-pyjwt==2.9.0
+pyflakes==3.2.0
+pygithub==2.4.0
+pyjwt==2.10.0
 pynacl==1.5.0
 pyopenssl==24.2.1
 pyparsing==3.2.0
-pyrsistent==0.20.0
 python-dateutil==2.9.0.post0
-python-dxf==11.4.0
+python-dxf==12.1.0
 python-editor==1.0.4
-python-gitlab==3.13.0
+python-gitlab==4.13.0
 pytz==2024.2
-pyyaml==6.0.1
+pyyaml==6.0.2
 pyzmq==26.2.0
 readchar==4.2.1
-requests==2.32.2
+referencing==0.35.1
+requests==2.32.3
 requests-toolbelt==1.0.0
 requirements-parser==0.2.0+1
-responses==0.22.0
-roundrobin==0.0.4
+responses==0.25.3
+rfc3339-validator==0.1.4
+rpds-py==0.21.0
 rsa==4.7.2
-s3transfer==0.7.0
+s3transfer==0.10.4
 setuptools==70.3.0
 setuptools-scm==5.0.2
 six==1.16.0
 smmap==5.0.1
 strict-rfc3339==0.7
-toml==0.10.2
-tqdm==4.67.0
-types-awscrt==0.23.0
-types-s3transfer==0.10.3
-types-toml==0.10.8.20240310
+tqdm==4.67.1
+types-awscrt==0.23.1
+types-s3transfer==0.10.4
 typing_extensions==4.12.2
-tzlocal==2.1
+tzlocal==5.2
 uritemplate==4.1.1
-urllib3==1.26.19
-watchdog==2.1.9
+urllib3==1.26.20
+watchdog==5.0.3
 wcwidth==0.2.13
 werkzeug==3.0.6
 wheel==0.38.4
-wrapt==1.16.0
+wrapt==1.17.0
 www-authenticate==0.9.2
 xmltodict==0.14.2
 zope.event==5.0

diff --git a/requirements.dev.trans.txt b/requirements.dev.trans.txt
@@ -1,59 +1,56 @@
 blessed==1.20.0
 blinker==1.9.0
-botocore-stubs==1.35.61
+botocore-stubs==1.35.69
 brotli==1.1.0
 click==8.1.7
-colorama==0.4.4
+colorama==0.4.6
 configargparse==1.7
 docutils==0.16
 et_xmlfile==2.0.0
 flask==3.0.3
-flask-basicauth==0.2.0
 flask-cors==5.0.0
-geventhttpclient==2.3.1
+flask-login==0.6.3
+geventhttpclient==2.3.3
 gitdb==4.0.11
 google-auth-httplib2==0.2.0
 greenlet==3.1.1
 httplib2==0.22.0
-importlib-resources==5.13.0
 inquirer==2.10.1
 itsdangerous==2.2.0
-jdcal==1.4.1
 jinja2==3.1.4
-jsonschema==4.17.3
-jsonschema-spec==0.1.6
-jwcrypto==1.5.6
+jsonschema==4.23.0
+jsonschema-path==0.3.3
+jsonschema-specifications==2023.12.1
 lazy-object-proxy==1.10.0
 mccabe==0.7.0
-mypy-boto3-dynamodb==1.28.73
-mypy-boto3-ecr==1.28.45
-mypy-boto3-iam==1.28.79
-mypy-boto3-kms==1.28.37
-mypy-boto3-lambda==1.28.83
-mypy-boto3-s3==1.28.55
-mypy-boto3-sqs==1.28.82
-mypy-boto3-stepfunctions==1.28.36
-openapi-schema-validator==0.3.4
+mypy-boto3-dynamodb==1.35.60
+mypy-boto3-ecr==1.35.21
+mypy-boto3-iam==1.35.61
+mypy-boto3-kms==1.35.0
+mypy-boto3-lambda==1.35.68
+mypy-boto3-s3==1.35.69
+mypy-boto3-sqs==1.35.0
+mypy-boto3-stepfunctions==1.35.68
+openapi-schema-validator==0.6.2
 pathable==0.4.3
 psutil==6.1.0
-py-partiql-parser==0.3.3
-pycodestyle==2.9.1
-pyflakes==2.5.0
-pyjwt==2.9.0
+py-partiql-parser==0.5.6
+pycodestyle==2.12.1
+pyflakes==3.2.0
+pyjwt==2.10.0
 pynacl==1.5.0
 pyparsing==3.2.0
-pyrsistent==0.20.0
 python-editor==1.0.4
 pyzmq==26.2.0
 readchar==4.2.1
+referencing==0.35.1
 requests-toolbelt==1.0.0
-roundrobin==0.0.4
+rfc3339-validator==0.1.4
+rpds-py==0.21.0
 smmap==5.0.1
-toml==0.10.2
-tqdm==4.67.0
-types-awscrt==0.23.0
-types-s3transfer==0.10.3
-types-toml==0.10.8.20240310
+tqdm==4.67.1
+types-awscrt==0.23.1
+types-s3transfer==0.10.4
 uritemplate==4.1.1
 wcwidth==0.2.13
 www-authenticate==0.9.2