OCPBUGS-59197: Fix MOSB image deletion race during MOSC removal by umohnani8 · Pull Request #6348 · openshift/machine-config-operator

umohnani8 · 2026-07-29T19:05:47Z

Closes https://redhat.atlassian.net/browse/OCPBUGS-59197

- What I did
When a MachineOSConfig is deleted while nodes are actively rebasing, the node annotation safety check in deleteMOSBImage was gated on moscExists. Since the MOSC is already gone in this code path, moscExists is always false and the check is skipped entirely, allowing the image to be deleted from the registry while nodes are still using it.

Remove the moscExists gate and look up the pool directly from the MOSB's labels. If the pool exists, check node annotations before deleting. If the pool is also gone, proceed with deletion since no nodes could be using the image.

- How to verify it

Follow the steps in https://redhat.atlassian.net/browse/OCPBUGS-59197 and the race should no longer happen

- Description for the changelog
Fix MOSB image deletion race during MOSC removal

Summary by CodeRabbit

Bug Fixes
- Improved cleanup safeguards for build registry images.
- Images are no longer deleted when they are currently applied to or desired by a node.
- Added clearer handling when the associated machine pool cannot be retrieved, including validation of required labeling and better error reporting.

openshift-merge-bot · 2026-07-29T19:05:49Z

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

openshift-ci-robot · 2026-07-29T19:05:54Z

@umohnani8: This pull request references Jira Issue OCPBUGS-59197, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (5.0.0) matches configured target version for branch (5.0.0)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @sergiordlr

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Closes https://redhat.atlassian.net/browse/OCPBUGS-59197

- What I did
When a MachineOSConfig is deleted while nodes are actively rebasing, the node annotation safety check in deleteMOSBImage was gated on moscExists. Since the MOSC is already gone in this code path, moscExists is always false and the check is skipped entirely, allowing the image to be deleted from the registry while nodes are still using it.

Remove the moscExists gate and look up the pool directly from the MOSB's labels. If the pool exists, check node annotations before deleting. If the pool is also gone, proceed with deletion since no nodes could be using the image.

- How to verify it

Follow the steps in https://redhat.atlassian.net/browse/OCPBUGS-59197 and the race should no longer happen

- Description for the changelog
Fix MOSB image deletion race during MOSC removal

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

coderabbitai · 2026-07-29T19:07:45Z

Walkthrough

deleteMOSBImage now derives the target MachineConfigPool from the MachineOSBuild label, handles pool lookup errors explicitly, and checks node image annotations before deletion.

Changes

MachineOSBuild image deletion

Layer / File(s)	Summary
Pool lookup and node image guards `pkg/controller/build/reconciler.go`	`deleteMOSBImage` validates the target pool label, retrieves the pool before scanning node image annotations, and reports unexpected pool lookup errors.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: proietfb, sergiordlr, eric200428

🚥 Pre-merge checks | ✅ 15

✅ Passed checks (15 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	Only pkg/controller/build/reconciler.go changed, and it contains no Ginkgo test declarations; no test names were added or modified.
Test Structure And Quality	✅ Passed	PR only changes pkg/controller/build/reconciler.go; no Ginkgo tests were added or modified, so the test-structure checklist is not applicable.
Microshift Test Compatibility	✅ Passed	Only pkg/controller/build/reconciler.go changed; no new Ginkgo e2e tests were added.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	Only pkg/controller/build/reconciler.go changed; no _test.go or Ginkgo e2e specs were added or modified, so SNO test compatibility is unaffected.
Topology-Aware Scheduling Compatibility	✅ Passed	Change is limited to controller image-deletion logic and label lookup; no manifests, replicas, affinities, nodeSelectors, or topology-aware scheduling constraints were added.
Ote Binary Stdout Contract	✅ Passed	Touched code is controller logic only; no main/init/TestMain/suite setup stdout writes were added, and deleteMOSBImage logs via klog.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	No Ginkgo/e2e tests were added or changed; only controller logic in reconciler.go was modified.
No-Weak-Crypto	✅ Passed	The PR only changes deleteMOSBImage image-deletion logic; there are no weak ciphers, custom crypto, or secret/token comparisons in the touched code.
Container-Privileges	✅ Passed	PR only changes Go reconciler logic; no Kubernetes manifests or securityContext settings were added or modified.
No-Sensitive-Data-In-Logs	✅ Passed	The PR only changes deleteMOSBImage logic; no new or modified logging statements were introduced.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: fixing a MOSB image deletion race during MachineOSConfig removal.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

umohnani8 · 2026-07-29T19:07:58Z

/pipeline required

openshift-merge-bot · 2026-07-29T19:08:01Z

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-ovn
/test e2e-aws-ovn-upgrade
/test e2e-gcp-op-ocl-part1
/test e2e-gcp-op-ocl-part2
/test e2e-gcp-op-part1
/test e2e-gcp-op-part2
/test e2e-gcp-op-single-node
/test e2e-hypershift

openshift-ci-robot · 2026-07-29T19:11:56Z

@umohnani8: This pull request references Jira Issue OCPBUGS-59197, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (5.0.0) matches configured target version for branch (5.0.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @sergiordlr

Details

In response to this:

Closes https://redhat.atlassian.net/browse/OCPBUGS-59197

- What I did
When a MachineOSConfig is deleted while nodes are actively rebasing, the node annotation safety check in deleteMOSBImage was gated on moscExists. Since the MOSC is already gone in this code path, moscExists is always false and the check is skipped entirely, allowing the image to be deleted from the registry while nodes are still using it.

Remove the moscExists gate and look up the pool directly from the MOSB's labels. If the pool exists, check node annotations before deleting. If the pool is also gone, proceed with deletion since no nodes could be using the image.

- How to verify it

Follow the steps in https://redhat.atlassian.net/browse/OCPBUGS-59197 and the race should no longer happen

- Description for the changelog
Fix MOSB image deletion race during MOSC removal

Summary by CodeRabbit

Bug Fixes

Improved cleanup safeguards for build registry images.

Images are no longer deleted when they are currently applied to or desired by a node.

Added clearer handling when the associated machine pool cannot be retrieved.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/controller/build/reconciler.go`:
- Around line 949-950: Update deleteMOSBImage to validate mosb.ObjectMeta.Labels
and TargetMachineConfigPoolLabelKey before calling machineConfigPoolLister.Get;
when the label is nil, missing, or empty, fail closed and do not enter the
deleted-pool handling or delete the image. Preserve that behavior only for a
non-empty pool name whose lookup returns NotFound.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 63dd120b-8c74-4506-8562-cc61bdb1af21

📥 Commits

Reviewing files that changed from the base of the PR and between dcc57bd and a5b4d12.

📒 Files selected for processing (1)

pkg/controller/build/reconciler.go

isabella-janssen · 2026-07-29T19:50:58Z

/test tls-pqc-readiness

Ignore this test, I'm just trying to show it's stable across PRs before making it required.

When a MachineOSConfig is deleted while nodes are actively rebasing, the node annotation safety check in deleteMOSBImage was gated on moscExists. Since the MOSC is already gone in this code path, moscExists is always false and the check is skipped entirely, allowing the image to be deleted from the registry while nodes are still using it. Remove the moscExists gate and look up the pool directly from the MOSB's labels. If the pool exists, check node annotations before deleting. If the pool is also gone, proceed with deletion since no nodes could be using the image. Signed-off-by: Urvashi <umohnani@redhat.com>

umohnani8 · 2026-07-30T13:18:22Z

/pipeline required

openshift-merge-bot · 2026-07-30T13:18:27Z

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-ovn
/test e2e-aws-ovn-upgrade
/test e2e-gcp-op-ocl-part1
/test e2e-gcp-op-ocl-part2
/test e2e-gcp-op-part1
/test e2e-gcp-op-part2
/test e2e-gcp-op-single-node
/test e2e-hypershift
/test tls-pqc-readiness

umohnani8 · 2026-07-30T18:51:54Z

/retest

umohnani8 · 2026-08-06T19:46:32Z

Pre-merge tested:

Environment:

OCP version: 5.0.0-0-2026-08-06-153257-test-ci-ln-gv331ft-latest
Platform: AWS (3 master, 3 worker)

Steps:

Created a push secret and MachineOSConfig for the worker pool to enable OCL with the internal registry.
Waited for the MachineOSBuild to succeed.

$ oc get machineosbuild -o wide
NAME                                      PREPARED   BUILDING   SUCCEEDED   INTERRUPTED   FAILED
worker-894be7259942189f5e3df25c806ac945   False      False      True        False         False

Confirmed a worker node had the desiredImage annotation set (actively rebasing).

$ oc get node/ip-10-0-51-21.ec2.internal -o jsonpath='{.metadata.annotations.machineconfiguration\.openshift\.io/desiredImage}'
image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/ocl-worker-image@sha256:9238f937...

Started capturing machine-os-builder logs, then deleted the MachineOSConfig while the node was still rebasing.

$ oc logs deployment/machine-os-builder -n openshift-machine-config-operator -f > /tmp/mob-logs.txt &
$ oc delete machineosconfig worker

Key result — the build reconciler detected the node annotation and refused to delete the image:

I0806 18:10:35.799815  reconciler.go:204] Removing MachineOSBuild(s) associated with non-existent MachineOSConfig worker
I0806 18:10:35.831996  reconciler.go:1173] Deleting MachineOSBuild "worker-894be7259942189f5e3df25c806ac945"
W0806 18:10:35.844966  reconciler.go:1097] Image ...@sha256:9238f937... is currently applied on a node or desired by a node, will not delete

Confirmed the image was preserved in the registry after MOSC deletion (HTTP 200 on both tag and digest lookups from inside the cluster).
Confirmed the node still had the desiredImage annotation after MOSC deletion.

$ oc get node/ip-10-0-51-21.ec2.internal -o jsonpath='{.metadata.annotations.machineconfiguration\.openshift\.io/desiredImage}'
image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/ocl-worker-image@sha256:9238f937...

MCP rolled back normally and MCO was not degraded.

$ oc get co machine-config -o jsonpath='{.status.conditions[?(@.type=="Degraded")].status}'
False

Summary: With the fix, deleteMOSBImage looks up the pool directly from the MOSB's target-machine-config-pool label instead of gating on moscExists. When the MOSC was deleted while a node was rebasing, the reconciler found the node's annotation matched the MOSB image and refused to delete it. Without this fix, moscExists would be false (MOSC already deleted), causing the safety check to be skipped and the image deleted while the node was still using it.

/verified by @umohnani8 and CI

openshift-ci-robot · 2026-08-06T19:46:44Z

@umohnani8: This PR has been marked as verified by @umohnani8 and CI.

Details

In response to this:

Pre-merge tested:

Environment:

OCP version: 5.0.0-0-2026-08-06-153257-test-ci-ln-gv331ft-latest

Platform: AWS (3 master, 3 worker)

Steps:

Created a push secret and MachineOSConfig for the worker pool to enable OCL with the internal registry.

Waited for the MachineOSBuild to succeed.
$ oc get machineosbuild -o wide
NAME                                      PREPARED   BUILDING   SUCCEEDED   INTERRUPTED   FAILED
worker-894be7259942189f5e3df25c806ac945   False      False      True        False         False
Confirmed a worker node had the desiredImage annotation set (actively rebasing).
$ oc get node/ip-10-0-51-21.ec2.internal -o jsonpath='{.metadata.annotations.machineconfiguration\.openshift\.io/desiredImage}'
image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/ocl-worker-image@sha256:9238f937...
Started capturing machine-os-builder logs, then deleted the MachineOSConfig while the node was still rebasing.
$ oc logs deployment/machine-os-builder -n openshift-machine-config-operator -f > /tmp/mob-logs.txt &
$ oc delete machineosconfig worker
Key result — the build reconciler detected the node annotation and refused to delete the image:
I0806 18:10:35.799815  reconciler.go:204] Removing MachineOSBuild(s) associated with non-existent MachineOSConfig worker
I0806 18:10:35.831996  reconciler.go:1173] Deleting MachineOSBuild "worker-894be7259942189f5e3df25c806ac945"
W0806 18:10:35.844966  reconciler.go:1097] Image ...@sha256:9238f937... is currently applied on a node or desired by a node, will not delete
Confirmed the image was preserved in the registry after MOSC deletion (HTTP 200 on both tag and digest lookups from inside the cluster).

Confirmed the node still had the desiredImage annotation after MOSC deletion.
$ oc get node/ip-10-0-51-21.ec2.internal -o jsonpath='{.metadata.annotations.machineconfiguration\.openshift\.io/desiredImage}'
image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/ocl-worker-image@sha256:9238f937...
MCP rolled back normally and MCO was not degraded.
$ oc get co machine-config -o jsonpath='{.status.conditions[?(@.type=="Degraded")].status}'
False
Summary: With the fix, deleteMOSBImage looks up the pool directly from the MOSB's target-machine-config-pool label instead of gating on moscExists. When the MOSC was deleted while a node was rebasing, the reconciler found the node's annotation matched the MOSB image and refused to delete it. Without this fix, moscExists would be false (MOSC already deleted), causing the safety check to be skipped and the image deleted while the node was still using it.

/verified by @umohnani8 and CI

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

isabella-janssen

/lgtm

openshift-merge-bot · 2026-08-06T19:49:25Z

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-ovn
/test e2e-aws-ovn-upgrade
/test e2e-gcp-op-ocl-part1
/test e2e-gcp-op-ocl-part2
/test e2e-gcp-op-part1
/test e2e-gcp-op-part2
/test e2e-gcp-op-single-node
/test e2e-hypershift
/test tls-pqc-readiness

openshift-ci · 2026-08-06T19:49:37Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: isabella-janssen, umohnani8

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [isabella-janssen,umohnani8]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-merge-bot · 2026-08-06T22:17:42Z

/retest-required

Remaining retests: 0 against base HEAD d690173 and 2 for PR HEAD b5f69a5 in total

umohnani8 · 2026-08-07T14:51:42Z

/retest

openshift-merge-bot · 2026-08-07T17:30:42Z

/retest-required

Remaining retests: 0 against base HEAD f6aefa6 and 1 for PR HEAD b5f69a5 in total

openshift-merge-bot · 2026-08-08T09:32:01Z

/retest-required

Remaining retests: 0 against base HEAD 8499975 and 0 for PR HEAD b5f69a5 in total

openshift-merge-bot · 2026-08-08T11:31:46Z

/hold

Revision b5f69a5 was retested 3 times: holding

umohnani8 · 2026-08-11T14:23:28Z

/hold cancel
/retest

openshift-ci · 2026-08-11T18:36:36Z

@umohnani8: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/perfscale-control-plane-6nodes	`b5f69a5`	link	false	`/test perfscale-control-plane-6nodes`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot · 2026-08-11T18:40:06Z

@umohnani8: Jira Issue Verification Checks: Jira Issue OCPBUGS-59197
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-59197 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

Closes https://redhat.atlassian.net/browse/OCPBUGS-59197

- What I did
When a MachineOSConfig is deleted while nodes are actively rebasing, the node annotation safety check in deleteMOSBImage was gated on moscExists. Since the MOSC is already gone in this code path, moscExists is always false and the check is skipped entirely, allowing the image to be deleted from the registry while nodes are still using it.

Remove the moscExists gate and look up the pool directly from the MOSB's labels. If the pool exists, check node annotations before deleting. If the pool is also gone, proceed with deletion since no nodes could be using the image.

- How to verify it

Follow the steps in https://redhat.atlassian.net/browse/OCPBUGS-59197 and the race should no longer happen

- Description for the changelog
Fix MOSB image deletion race during MOSC removal

Summary by CodeRabbit

Bug Fixes

Improved cleanup safeguards for build registry images.

Images are no longer deleted when they are currently applied to or desired by a node.

Added clearer handling when the associated machine pool cannot be retrieved, including validation of required labeling and better error reporting.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-merge-robot · 2026-08-13T03:13:47Z

Fix included in release 5.0.0-0.nightly-2026-08-12-201610

openshift-ci Bot requested review from proietfb and sergiordlr July 29, 2026 19:05

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 29, 2026

coderabbitai Bot reviewed Jul 29, 2026

View reviewed changes

Comment thread pkg/controller/build/reconciler.go Outdated

umohnani8 force-pushed the OCPBUGS-59197-fix-mosb-image-deletion-race branch from a5b4d12 to eaa8a7c Compare July 30, 2026 13:16

umohnani8 force-pushed the OCPBUGS-59197-fix-mosb-image-deletion-race branch from eaa8a7c to b5f69a5 Compare July 30, 2026 13:17

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Aug 6, 2026

isabella-janssen reviewed Aug 6, 2026

View reviewed changes

openshift-ci Bot assigned isabella-janssen Aug 6, 2026

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Aug 6, 2026

openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 8, 2026

openshift-ci Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 11, 2026

openshift-merge-bot Bot merged commit dcb37f6 into openshift:main Aug 11, 2026
18 of 19 checks passed

Uh oh!

Conversation

umohnani8 commented Jul 29, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

openshift-merge-bot Bot commented Jul 29, 2026

Uh oh!

openshift-ci-robot commented Jul 29, 2026

Uh oh!

coderabbitai Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

umohnani8 commented Jul 29, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 29, 2026

Uh oh!

openshift-ci-robot commented Jul 29, 2026

Summary by CodeRabbit

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

isabella-janssen commented Jul 29, 2026

Uh oh!

umohnani8 commented Jul 30, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 30, 2026

Uh oh!

umohnani8 commented Jul 30, 2026

Uh oh!

umohnani8 commented Aug 6, 2026

Uh oh!

openshift-ci-robot commented Aug 6, 2026

Uh oh!

isabella-janssen left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-merge-bot Bot commented Aug 6, 2026

Uh oh!

openshift-ci Bot commented Aug 6, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 6, 2026

Uh oh!

umohnani8 commented Aug 7, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 7, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 8, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 8, 2026

Uh oh!

umohnani8 commented Aug 11, 2026

Uh oh!

openshift-ci Bot commented Aug 11, 2026

Uh oh!

Uh oh!

openshift-ci-robot commented Aug 11, 2026

Summary by CodeRabbit

Uh oh!

openshift-merge-robot commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

umohnani8 commented Jul 29, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 29, 2026 •

edited

Loading