Skip to content

OCPBUGS-97602: Use glance image with stable name to avoid race condition#81445

Open
winiciusallan wants to merge 1 commit into
openshift:mainfrom
shiftstack:change-rhcos-upload
Open

OCPBUGS-97602: Use glance image with stable name to avoid race condition#81445
winiciusallan wants to merge 1 commit into
openshift:mainfrom
shiftstack:change-rhcos-upload

Conversation

@winiciusallan

@winiciusallan winiciusallan commented Jul 3, 2026

Copy link
Copy Markdown
Member

In the current script, when two jobs are triggered in close succession, there is a risk that one of them might overwrite the image name the other was using, causing one of the tests to fail.

This PR changes the script to use a stable image name.

Summary by CodeRabbit

This updates the OpenStack RHCOS image pre-upload flow used by OpenShift CI so jobs share a stable Glance image name instead of racing to rename or replace versioned images. In practice, the script now derives the image name from the installer’s embedded RHCOS stream version, creates the image directly under that final name when needed, and cleans up duplicate entries before continuing.

The accompanying CI config documentation was updated to describe the new naming behavior, so downstream jobs and maintainers have the correct expectation for how the shared RHCOS image name is formed.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 3, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@winiciusallan: This pull request references Jira Issue OCPBUGS-97602, which is invalid:

  • expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

In the current script, when two jobs are triggered in close succession, there is a risk that one of them might overwrite the image name the other was using, causing one of the tests to fail.

This PR changes the script to use a stable image name.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Jul 3, 2026
@coderabbitai

coderabbitai Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Walkthrough

This PR changes an OpenStack RHCOS image configuration script to derive the target image name from the installer's stream JSON version instead of the OCP minor version, and simplifies the upload logic from sha256-comparison/rename-based promotion to a straightforward existence check followed by direct image creation. Documentation is updated to match.

Changes

RHCOS Image Naming and Upload Flow

Layer / File(s) Summary
Version derivation and image naming
ci-operator/step-registry/openstack/conf/rhcosimage/openstack-conf-rhcosimage-commands.sh
Removes OCP_VERSION extraction from openshift-install version and builds IMAGE_NAME from OPENSTACK_RHCOS_IMAGE_NAME plus the stream JSON's IMAGE_VERSION, updating log messages accordingly.
Existence-based upload and direct image creation
ci-operator/step-registry/openstack/conf/rhcosimage/openstack-conf-rhcosimage-commands.sh
Replaces sha256-property comparison with an EXISTING_COUNT check to skip or proceed with upload, and replaces the -new/-old rename/promotion flow with a direct openstack image create under the final IMAGE_NAME, capturing the image ID and cleaning up the working directory.
Documentation update
ci-operator/step-registry/openstack/conf/rhcosimage/openstack-conf-rhcosimage-ref.yaml
Updates the OPENSTACK_RHCOS_IMAGE_NAME description to reflect automatic appending of the RHCOS stream version and immutable per-build Glance images.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Related PRs: None identified.

Suggested labels: ci-operator, openstack, documentation

Suggested reviewers: None identified.

🐰 A rabbit hopped through streams and names,
No more old-new promotion games,
One image built, its ID recorded,
Version tagged and duly sorted,
Docs updated, script refined—
a cleaner path for RHCOS to find.

🚥 Pre-merge checks | ✅ 5 | ❌ 10

❌ Failed checks (10 inconclusive)

Check name Status Explanation Resolution
Stable And Deterministic Test Names ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Test Structure And Quality ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Microshift Test Compatibility ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Single Node Openshift (Sno) Test Compatibility ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Topology-Aware Scheduling Compatibility ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Ote Binary Stdout Contract ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Ipv6 And Disconnected Network Test Compatibility ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
No-Weak-Crypto ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Container-Privileges ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
No-Sensitive-Data-In-Logs ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: using a stable Glance image name to prevent a race condition.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot requested review from gryf and stephenfin July 3, 2026 12:29
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@winiciusallan: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-openshift-cluster-api-provider-openstack-main-e2e-openstack-capi-techpreview openshift/cluster-api-provider-openstack presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-openstack-release-5.1-e2e-openstack-capi-techpreview openshift/cluster-api-provider-openstack presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-openstack-release-5.0-e2e-openstack-capi-techpreview openshift/cluster-api-provider-openstack presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-openstack-release-4.23-e2e-openstack-capi-techpreview openshift/cluster-api-provider-openstack presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-openstack-release-4.22-e2e-openstack-capi-techpreview openshift/cluster-api-provider-openstack presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-openstack-release-4.21-e2e-openstack-capi-techpreview openshift/cluster-api-provider-openstack presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-openstack-release-4.20-e2e-techpreview openshift/cluster-api-provider-openstack presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-openstack-release-4.19-e2e-techpreview openshift/cluster-api-provider-openstack presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-openstack-release-4.18-e2e-techpreview openshift/cluster-api-provider-openstack presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-openstack-release-4.17-e2e-techpreview openshift/cluster-api-provider-openstack presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-openstack-release-4.16-e2e-techpreview openshift/cluster-api-provider-openstack presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-openstack-release-4.15-e2e-techpreview openshift/cluster-api-provider-openstack presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-openstack-release-4.14-e2e-openstack openshift/cluster-api-provider-openstack presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-openstack-release-4.14-e2e-openstack-parallel openshift/cluster-api-provider-openstack presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-openstack-release-4.14-e2e-openstack-operator openshift/cluster-api-provider-openstack presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-openstack-release-4.14-e2e-openstack-proxy openshift/cluster-api-provider-openstack presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-openstack-release-4.14-e2e-upgrade openshift/cluster-api-provider-openstack presubmit Registry content changed
pull-ci-openshift-cluster-cloud-controller-manager-operator-release-4.11-e2e-openstack-ccm openshift/cluster-cloud-controller-manager-operator presubmit Registry content changed
pull-ci-openshift-cluster-cloud-controller-manager-operator-release-4.10-e2e-openstack-ccm openshift/cluster-cloud-controller-manager-operator presubmit Registry content changed
pull-ci-openshift-cluster-cloud-controller-manager-operator-release-4.9-e2e-openstack-ccm openshift/cluster-cloud-controller-manager-operator presubmit Registry content changed
pull-ci-openshift-cluster-cloud-controller-manager-operator-main-e2e-openstack-ovn openshift/cluster-cloud-controller-manager-operator presubmit Registry content changed
pull-ci-openshift-cluster-cloud-controller-manager-operator-release-5.1-e2e-openstack-ovn openshift/cluster-cloud-controller-manager-operator presubmit Registry content changed
pull-ci-openshift-cluster-cloud-controller-manager-operator-release-5.0-e2e-openstack-ovn openshift/cluster-cloud-controller-manager-operator presubmit Registry content changed
pull-ci-openshift-cluster-cloud-controller-manager-operator-release-4.23-e2e-openstack-ovn openshift/cluster-cloud-controller-manager-operator presubmit Registry content changed
pull-ci-openshift-cluster-cloud-controller-manager-operator-release-4.22-e2e-openstack-ovn openshift/cluster-cloud-controller-manager-operator presubmit Registry content changed

A total of 1161 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here
Prior to this PR being merged, you will need to either run and acknowledge or opt to skip these rehearsals.

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@mandre mandre left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice one.
/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 3, 2026
@openshift-ci

openshift-ci Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mandre, winiciusallan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 3, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@ci-operator/step-registry/openstack/conf/rhcosimage/openstack-conf-rhcosimage-commands.sh`:
- Around line 70-73: The skip logic in the RHCOS image upload flow is too broad
and currently reuses any matching image instead of only a ready one. Update the
check in the image lookup/upload path to inspect the OpenStack image JSON status
via .[0].status and only skip when the existing image is active; for queued,
saving, or killed states, do not reuse it and instead refresh or fail so the
upload path can proceed correctly. Use the existing IMAGE_NAME/EXISTING_COUNT
handling in the openstack-conf-rhcosimage-commands.sh flow to locate the
condition.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 503b4181-a334-4ed3-ae6d-bde904d3791e

📥 Commits

Reviewing files that changed from the base of the PR and between a3b2e4d and 52361b6.

📒 Files selected for processing (2)
  • ci-operator/step-registry/openstack/conf/rhcosimage/openstack-conf-rhcosimage-commands.sh
  • ci-operator/step-registry/openstack/conf/rhcosimage/openstack-conf-rhcosimage-ref.yaml

Comment on lines +70 to +73
if [[ "$EXISTING_COUNT" -eq 1 ]]; then
echo "RHCOS image '${IMAGE_NAME}' already exists. Skipping upload."
else
echo "RHCOS image '${IMAGE_NAME}' needs to be uploaded (current sha256: '${CURRENT_SHA256}', expected: '${UNCOMPRESSED_SHA256}')"
echo "RHCOS image '${IMAGE_NAME}' not found. Uploading..."

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

sed -n '1,180p' ci-operator/step-registry/openstack/conf/rhcosimage/openstack-conf-rhcosimage-commands.sh

Repository: openshift/release

Length of output: 2105


🏁 Script executed:

rg -n "openstack image list -f json|\\.Status|\\.status|image list -f json" ci-operator/step-registry/openstack -S

Repository: openshift/release

Length of output: 2105


🏁 Script executed:

python3 - <<'PY'
import json, re, sys, pathlib
p = pathlib.Path("ci-operator/step-registry/openstack/conf/rhcosimage/openstack-conf-rhcosimage-commands.sh")
text = p.read_text()
for m in re.finditer(r'openstack image list -f json.*?jq -r \'([^\']+)\'', text, re.S):
    print(m.group(1))
PY

Repository: openshift/release

Length of output: 2105


🌐 Web query:

OpenStackClient openstack image list -f jsonJSON field names status casingStatus status docs

💡 Result:

When using the OpenStackClient command openstack image list -f json, the field name for the image status is lowercase status [1][2][3]. It is not capitalized as Status [1][3][4]. The OpenStack Image Service API (Glance), which backs this command, consistently uses the lowercase status key in its JSON response objects [2][5][3]. Similarly, the internal Python-OpenStackClient code that formats these results maps the field to status [1], and the OpenStack SDK resource definition for an image also uses status as the attribute name [4].

Citations:


Only skip when the existing image is active. openstack image list -f json uses the lowercase status field, so check .[0].status here and fail/refresh on queued, saving, or killed images instead of reusing them.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/openstack/conf/rhcosimage/openstack-conf-rhcosimage-commands.sh`
around lines 70 - 73, The skip logic in the RHCOS image upload flow is too broad
and currently reuses any matching image instead of only a ready one. Update the
check in the image lookup/upload path to inspect the OpenStack image JSON status
via .[0].status and only skip when the existing image is active; for queued,
saving, or killed states, do not reuse it and instead refresh or fail so the
upload path can proceed correctly. Use the existing IMAGE_NAME/EXISTING_COUNT
handling in the openstack-conf-rhcosimage-commands.sh flow to locate the
condition.

@winiciusallan

winiciusallan commented Jul 3, 2026

Copy link
Copy Markdown
Member Author

/pj-rehearse

rehearsing a few jobs first.

@winiciusallan

Copy link
Copy Markdown
Member Author

/pj-rehearse

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@winiciusallan: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@winiciusallan

Copy link
Copy Markdown
Member Author

/pj-rehearse pull-ci-openshift-origin-release-4.22-e2e-openstack-ovn

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@winiciusallan: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@winiciusallan

Copy link
Copy Markdown
Member Author

/pj-rehearse pull-ci-openshift-origin-release-4.22-e2e-openstack-ovn

Error related with openshift/openstack-test@ecdba93, however the fix is now merged on 4.22. Rehearsing again

{  fail [github.com/openshift/openstack-test/test/extended/openstack/machine.go:142]: SGs not matching for "1isk4nnp-b6910-9svrf-master-0"
Expected
    <map[string]struct {} | len:3>: 
        1isk4nnp-b6910-9svrf-master: {}
        lb-sg-180611b6-734d-452e-a2ba-72d4410a3492-e2e-udn-services-6854-udn-service: {}
        lb-sg-d79cf18d-9623-4005-a0a4-7b9041ad120b-e2e-udn-services-2565-udn-service: {}
to equal
    <map[string]struct {} | len:1>: 
        1isk4nnp-b6910-9svrf-master: {}

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@winiciusallan: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci

openshift-ci Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

@winiciusallan: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants