Skip to content

Upgrade certifi, pip, uv, ca-certificates, and pixi for the eMudhra/emSign CA incident #3195

Closed

Description

@juenglin

The Python security advisory Information for Python users regarding eMudhra / emSign CA certificate security incident asks users to upgrade immediately to at least certifi 2026.10.19, pip 26.2.2, uv 0.13.4, sigstore 4.5.1, conda 26.9.4, and pixi 0.82.1. Older versions still trust the compromised eMudhra intermediate CAs, which exposes TLS clients (including connections to pypi.org and python.org) to man-in-the-middle attacks. For pip in particular, versions before 26.2.2 always load their vendored certifi bundle in addition to the system trust store, so refreshing the system store alone is not enough.

On main (checked at afa6249), our pixi lockfiles and several CI pins are below these floors, and many CI jobs install packages with an older pip. Please fix these on main first; the fix will then be merged into the release branches.

Pixi lockfiles

All of these are conda-forge packages:

Lockfile certifi pip uv ca-certificates
pixi.lock - - - 2026.7.22
cuda_pathfinder/pixi.lock 2026.7.22 26.2.1 0.12.23 2026.7.22
cuda_bindings_13/pixi.lock 2026.7.22 26.2.1 0.12.23 2026.7.22
cuda_core/pixi.lock 2026.7.22 26.2.1 0.12.21 2026.7.22
benchmarks/cuda_bindings/pixi.lock - - 0.12.23 2026.7.22
benchmarks/cuda_core/pixi.lock - - 0.12.23 2026.7.22
  • certifi and pip appear only in the docs environments.
  • uv appears in the host_packages of the pixi-build source packages (cuda-pathfinder, cuda-bindings, cuda-core).
  • ca-certificates appears in every environment and source-package host environment. It is not on the advisory's list, but it is the conda-forge counterpart of the system certificate store that the advisory says to refresh, and 2026.7.22 predates the incident.

conda-forge already has fixed builds: certifi 2026.10.19, pip 26.2.2, uv 0.13.5, and ca-certificates 2026.10.19. None of the lockfiles contain sigstore or conda.

CI pins

  • ci/pixi-version.env sets PIXI_VERSION="v0.73.0", which .github/actions/setup-pixi installs. The advisory requires pixi 0.82.1 or later.
  • Related fixes announced later in the same thread:
    • .github/workflows/build-wheel.yml uses pypa/cibuildwheel v4.3.1. cibuildwheel 4.3.3 switched to fixed manylinux images, and 4.3.4 extends the fix to more targets and warns when uv is too old. We don't override the manylinux images, so the bump picks up the fixed ones. The uv check only warns, so look for that warning in the first build logs after the bump.
    • .github/workflows/release.yml and .github/workflows/release-cuda-pathfinder.yml use pypa/gh-action-pypi-publish v1.14.2. v1.14.3 contains the fix.

pip in CI jobs

Many jobs install from PyPI with the pip that came with their Python: from actions/setup-python, python -m venv, or the runner's preinstalled Python. setup-python's current 3.11 and 3.12 builds date from Oct 1. Only its 3.13.17, 3.14.9, and 3.15.1 builds (Oct 21-22) are newer than pip 26.2.2, and a runner whose tool cache already has a matching version can still use an older build. The existing upgrade steps ("pip>=25.1" in test-wheel-linux.yml and test-wheel-windows.yml, "pip>=25.3" in coverage.yml and test-sdist-*.yml, and --upgrade pip in pre-commit.yml) download the new pip with the old one, and raising those floors wouldn't change that. The advisory's method is to install the fixed pip by direct URL with its hash:

python -m pip install "pip @ https://files.pythonhosted.org/packages/3f/06/3e4420302363fbb71be3c7f35ce042f54bfca330e3e66b3a2e3c4cca579f/pip-26.2.2-py3-none-any.whl#sha256=fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70"

Add one shared step that runs this before the first pip use in each job, and replace the existing upgrade steps with it. Jobs that use pip:

  • build-wheel.yml: pip wheel builds the cuda-pathfinder and cuda-python wheels (on Python 3.12), plus pip install twine and the wheel test installs.
  • test-wheel-linux.yml, test-wheel-windows.yml, test-sdist-linux.yml, and test-sdist-windows.yml.
  • coverage.yml, including inside each python -m venv .venv.
  • pre-commit.yml, plus lychee.yml and the link check in build-docs.yml, which install pre-commit through .github/actions/lychee-links.
  • ci-nightly.yml, which uses the runner's preinstalled Python.
  • release.yml and release-cuda-pathfinder.yml: the post-publish install checks.

The pre-commit cache key in pre-commit.yml and lychee.yml changes only with .pre-commit-config.yaml or the Python version, so cached hook environments built with an older pip keep being reused. Change the key once the new step lands. pre-commit creates hook environments with virtualenv, and virtualenv 21.14.13 or later bundles pip 26.2.2.

Checked, no change needed: the astral-sh/setup-uv steps (no version input; the latest bandit run installed uv 0.13.5). The Docs job's miniforge-version: latest now resolves to Miniforge 26.7.2-1 (Oct 21), a rebuild that forces the fixed CA bundles; conda 26.9.4's only change is requiring them. No pixi manifest caps pip, uv, or the pixi version, and no install cooldowns are configured. cuda_bindings_12/ has no pixi workspace and is not built by CI on main.

Constraints from the lockfile maintenance pause

ci/README.md suspends automated lockfile refreshes because of prefix-dev/pixi#7215 and prefix-dev/pixi#7000, and leaves the pinned pixi version and committed lockfiles unchanged during the pause. For this update:

Release branches

After the fix is on main, it will be merged into 13.4.x and then 12.9.x. The pixi changes won't merge cleanly into either branch:

  • 13.4.x pins cibuildwheel v4.2.0 and gh-action-pypi-publish v1.14.2. It has no ci/pixi-version.env (it pins pixi v0.73.0 in ci-pixi-source-test.yml), and its six lockfiles are also below the floors.
  • 12.9.x pins cibuildwheel v4.3.1 and gh-action-pypi-publish v1.14.2, and has no pixi workspaces. build-wheel.yml on main downloads the CUDA 12 cuda-bindings wheels from that branch's CI runs.

Checklist

  • Upgrade certifi, pip, uv, and ca-certificates in all six lockfiles to at least the versions above.
  • In the same commit, bump PIXI_VERSION in ci/pixi-version.env to v0.82.1 or later (after the --frozen check), and record the exception in ci/README.md.
  • Bump pypa/cibuildwheel to v4.3.4 or later and pypa/gh-action-pypi-publish to v1.14.3 or later.
  • Install pip 26.2.2 from its hash-pinned URL before the first pip use in every CI job, replacing the existing pip upgrade steps.
  • After that, change the pre-commit cache key in pre-commit.yml and lychee.yml.
  • Merge the fix into 13.4.x, then 12.9.x.

Metadata

Metadata

Assignees

Labels

CI/CDCI/CD infrastructureP0High priority - Must do!dependenciesPull requests that update a dependency file

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions