Upgrade graph edge from `4.22.0-okd-scos.7` to `4.22.0-okd-scos.8` fails to create

[pskrbasu]

The upgrade verification job release-openshift-origin-installer-launch-aws-modern consistently fails when testing the upgrade path from 4.22.0-okd-scos.7 to 4.22.0-okd-scos.8, preventing the upgrade graph edge
from being created in the 4-scos-stable release stream.

Both releases are individually Accepted, and upgrades from older releases (scos.0 through scos.3) to scos.8 succeed. The issue appears specific to upgrades from scos.4 and later. scos.4 to scos.6 is understandable as it had a cri-o issue, but that was resolved in scos.7. So it was expected that scos.7 to future releases would be smooth.

Failed runs:

• https://prow.ci.openshift.org/view/gs/test-platform-results/logs/release-openshift-origin-installer-launch-aws-modern/2090443010234912768
• https://prow.ci.openshift.org/view/gs/test-platform-results/logs/release-openshift-origin-installer-launch-aws-modern/2090400105076101120

[JohnStarich]

I’m a novice on upgrade paths here, but I think there’s a similar busted graph edge from 4.22.0-okd-scos.4 to any next release.

[ProbstDJakob]

Yes, as @pskrbasu pointed out it starts with scos.4 up to (and including) scos.7. You can see what upgrades are successful/have failed (and why they failed if you click on the F/Failed text) when you go to the version you want to upgrade to: https://origin-release.ci.openshift.org/releasestream/4-scos-stable/release/4.22.0-okd-scos.8

[JohnStarich]

@ProbstDJakob Ah, maybe I misunderstood the scope from the title. It only says from .7 to .8, but it’s also busted on edges for .4 onwards (to all listed releases).

[ProbstDJakob]

That is because .4, .5, and .6 are known to fail because of a bug in CRI-O (see #2340), but .7 to .8 should have worked, but didn’t. Thus it is the only version mentioned in the title.

[JohnStarich]

Understood, thanks! I should have searched the history better 😄

[ProbstDJakob]

Or have read the description of this issue more carefully ;) but no problem

[rezemble]

upgrade edge from .8 to .9 looks fine, is this related?

[pskrbasu]

@rezemble No this is un-related. This was specific to scos.7 to scos.8

[pskrbasu]

Due to CI issues with our internal upgrade verification tests, the upgrade graph edge from 4.22.0-okd-scos.7 to 4.22.0-okd-scos.8 could not be created. This means the automatic upgrade path is not available between these two versions.

If you are currently on scos.7 and want to upgrade, the only option right now is to force the upgrade:

oc adm upgrade --force --to-image=quay.io/okd/scos-release:4.22.0-okd-scos.10

We have tested this ourselves, and several community members have also confirmed that the upgrade works fine via this path.

Upgrade edges are available from scos.8 onwards to the latest release, so once you are on scos.8 you will be able to upgrade normally going forward.

[risochras]

Due to CI issues with our internal upgrade verification tests, the upgrade graph edge from 4.22.0-okd-scos.7 to 4.22.0-okd-scos.8 could not be created. This means the automatic upgrade path is not available between these two versions.

If you are currently on scos.7 and want to upgrade, the only option right now is to force the upgrade:

oc adm upgrade --force --to-image=quay.io/okd/scos-release:4.22.0-okd-scos.10

We have tested this ourselves, and several community members have also confirmed that the upgrade works fine via this path.

Upgrade edges are available from scos.8 onwards to the latest release, so once you are on scos.8 you will be able to upgrade normally going forward.

Can one upgrade from 4.22.0-okd-scos.4 straight to 4.22.0-okd-scos.10 ? Just upgraded (with --force) from scos.4 to scos.7 and it went fine.