Skip to content

[CI] Pin the wheel-test torch to a shipped PG build - #4440

Open
he-yufeng wants to merge 1 commit into
mainfrom
fix/ci-wheel-torch-pg-pin
Open

he-yufeng wants to merge 1 commit into
mainfrom
fix/ci-wheel-torch-pg-pin

Conversation

@he-yufeng

@he-yufeng he-yufeng commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Fixes #4439.

The wheel test legs of CI are red on every PR right now. scripts/test_installation.sh installs torch unpinned, so PyPI's latest (2.14.1) lands in the test venv, while the wheel only ships PG builds for EP_TORCH_VERSIONS (currently up to 2.14.0). mooncake.pg then looks for mooncake.pg_2_14_1, and all six test_pg_* modules fail at import.

The wheel is already installed a few lines earlier in the same script, so it now enumerates the mooncake.pg_* modules the wheel ships and installs the newest matching torch. Wheels without PG builds (non-cuda variants, or any detection failure) keep the previous unpinned behavior, so the change cannot make a green run worse.

Module

  • CI/CD

Type of Change

  • Bug fix

How Has This Been Tested?

Test commands:

bash -n scripts/test_installation.sh
# detection snippet run against a stub mooncake package with five fake
# pg_*.so builds (2.11.0 .. 2.14.0), against one with no PG modules, and
# against an environment where import mooncake fails outright

Test results:

  • stub with five PG builds: resolves torch==2.14.0, the newest shipped variant
  • stub with no PG modules: empty result, falls back to unpinned pip install torch numpy
  • import failure: || true keeps set -e intact, same fallback
  • the end-to-end validation is this PR's own test-wheel-ubuntu legs, which should go green again

Checklist

  • I have performed a self-review of my own code
  • I have run pre-commit on the files changed in this PR and all hooks pass (shell script, verified with bash -n; no pre-commit hook covers it)
  • I have updated the documentation (not applicable, no behavior change for users)
  • I have added tests to prove my changes are effective (the CI pipeline itself is the test: this leg is red on every open PR today)

AI Assistance Disclosure

  • AI tools were used (specify below)

Kimi Code drafted the change; verification steps above were actually run as described, and the submitter has reviewed every changed line.

test_installation.sh installs torch unpinned, so a fresh PyPI release
(2.14.1) lands in the test env while the wheel only ships PG builds for
EP_TORCH_VERSIONS (up to 2.14.0). mooncake.pg then looks for
mooncake.pg_2_14_1, and all six test_pg_* modules fail at import on
every PR (#4439).

The wheel is already installed at that point, so enumerate the
mooncake.pg_* modules it ships and install the newest matching torch.
Wheels without PG builds (non-cuda) keep the previous unpinned
behavior.

Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
@he-yufeng

Copy link
Copy Markdown
Collaborator Author

Status update: the end-to-end validation the description was waiting for is in. Both test-wheel-ubuntu legs pass on this PR (py312 in 20m58s, py310 in 6m22s), while the same job fails on every other open PR with ImportError: Mooncake PG was not built against torch==2.14.1 (example: job 110676867503 on #4450).

Since this unblocks CI for the whole open queue, a review when someone has a moment would help. @Aionw you merged most of the recent CI fixes, maybe you have context on this one.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: test-wheel CI legs red on all PRs: unpinned pip install torch pulls 2.14.1, wheel only ships PG builds up to 2.14.0

1 participant