Fix quay-proxy ServiceAccount login username

[tusharjadhav3302]

Summary

Follow-up to #1933: fix write_pull_secret() so quay-proxy login works for ServiceAccount CI tokens.

#1933 correctly switched the default CI registry to quay-proxy.ci.openshift.org and added a podman login step. The username it used breaks auth for ServiceAccounts.

Problem

podman login "${CI_REGISTRY}" \
    --username "$(oc whoami)" \
    --password-stdin

For a CI ServiceAccount, oc whoami returns:

system:serviceaccount:openstack-k8s-operators:image-puller

The : characters corrupt HTTP Basic Auth password parsing on quay-proxy, so login fails with:

Error: logging into "quay-proxy.ci.openshift.org": invalid username/password
→ make: *** [Makefile:74: build_installer] Error 125

The CI token itself is fine (oc login and oc registry login succeed; the SA can see projects). Only the username format is wrong.

CI docs explicitly warn about this:
https://docs.ci.openshift.org/how-tos/use-registries-in-build-farm/

IMPORTANT: do NOT use -u="$(oc whoami)" for a ServiceAccount.
Use any colon-free username; the password must be the ServiceAccount token.

Change

File: utils.sh (write_pull_secret())

-            --username "$(oc --kubeconfig="$tmpkubeconfig" whoami)" \
+            --username "image-puller" \

Password is unchanged: still piped from oc whoami -t.

Impact

Without this, any non-OPENSHIFT_CI path that calls write_pull_secret() after #1933 fails at build_installer when authenticating to quay-proxy (e.g. ShiftStack/RHOSO Zuul jobs on serval70 using devscripts_ci_token).

Reproduction / validation

# BAD (current #1933 behavior) — expect fail
oc whoami -t | podman login quay-proxy.ci.openshift.org \
  -u "$(oc whoami)" --password-stdin --authfile /tmp/bad.json

# GOOD (this PR) — expect ok
oc whoami -t | podman login quay-proxy.ci.openshift.org \
  -u image-puller --password-stdin --authfile /tmp/good.json

Test plan

• Confirm CI token still logs into app.ci as the image-puller SA
• Confirm broken username form fails quay-proxy login
• Confirm -u image-puller succeeds
• Re-run a ShiftStack/RHOSO periodic (or equivalent) past 03_build_installer.sh / write_pull_secret

Related

• Regression from: NO-ISSUE: switch default CI registry to quay-proxy.ci.openshift.org #1933 (@andfasano)
• Independent of: OCPBUGS-100315: Fix baremetal registry and pull secret after #1922 #1937 (baremetal REGISTRY_CREDS / local registry — still needed separately)
• Docs: https://docs.ci.openshift.org/how-tos/use-registries-in-build-farm/
• QCI migration context: https://redhat-internal.slack.com/archives/CFUGK0K9R/p1785235182016299

Made with Cursor

[openshift-ci]

Hi @tusharjadhav3302. Thanks for your PR.

I'm waiting for a openshift-metal3 member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[andfasano]

/ok-to-test

[andfasano]

I'm not sure that quay-proxy manages all the user via the hard-coded image-puller, since the docs explicitly mention the needs of registering a rover group.

In that case, I'd suggest implementing a solution that could support both cases (humans via rover group and service accounts), ie something like (just a suggestion from Claude):

--username "$(oc --kubeconfig="$tmpkubeconfig" whoami | tr ':' '_')" \

[tusharjadhav3302]

Thanks @andfasano — updated.

Dropped the hardcoded image-puller username and switched to your suggestion:

--username "$(oc --kubeconfig="$tmpkubeconfig" whoami | tr ':' '_')"

That keeps human / Rover identities intact when whoami has no colons, and makes ServiceAccount names Basic Auth safe for quay-proxy (system:serviceaccount:… → system_serviceaccount_…). Password is still the token from oc whoami -t.

[dtantsur]

/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dtantsur

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [dtantsur]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tusharjadhav3302]

/retest

[tusharjadhav3302]

CI failure triage (e2e-metal-ipi-ovn-ipv4 / dualstack / ipv6)

Looked at the three failing Prow jobs. These do not appear related to this PR.

What this PR touches

write_pull_secret() only changes the quay-proxy podman login username (oc whoami | tr ':' '_'). That runs during cluster install / build_installer.

What the jobs actually did

On all three jobs, baremetalds-devscripts-setup succeeded (~1h14–1h16). There is no invalid username/password, no build_installer failure, and no quay-proxy login error in these logs. The failure is later, in baremetalds-e2e-test.

Job	Setup	Failed step
e2e-metal-ipi-ovn-ipv4	succeeded	baremetalds-e2e-test (~5h10m)
e2e-metal-ipi-ovn-dualstack	succeeded	baremetalds-e2e-test (~5h10m)
e2e-metal-ipi-ovn-ipv6	succeeded	baremetalds-e2e-test (~1h58m)

Failure details

ipv4 / dualstack — many [sig-storage] failures with the same symptom: pods cannot pull mirrored e2e test images from the local registry:

Failed to pull image "virthost.ostest.test.metalkube.org:5000/localimages/local-test-image:e2e-..."
manifest unknown
ErrImagePull / ImagePullBackOff

Examples: CSI hostpath/mock plugins, Secrets/EmptyDir/Subpath conformance pods timing out waiting for images.

ipv6 — suite was largely green (2058 pass, 9 informing, 1 blocking):

[sig-auth][Feature:OAuthServer] [Token Expiration]
Using a OAuth client with a non-default token max age ...
tokens that do not expire works as expected when using a token authorization flow

Conclusion

Install path (where this PR applies) worked. Failures are post-install e2e flakes / local-registry test-image pull issues (manifest unknown) and a single OAuth blocking flake on ipv6 — not caused by the colon-safe username change.

Do we do a re-run again ( consumes 5 hours ) or could we merge this fix without it?
cc : @dtantsur

[dtantsur]

/retest

Let's see if the pattern changes at least.

[tusharjadhav3302]

Manual validation (serval71)

Completed a manual validation of this PR on serval71, using the same credentials path our RHOSO jobs use for write_pull_secret.

What I ran

1. Loaded the job CI token and pointed at the build-farm API / registry:

   • CI_TOKEN from /home/zuul/src/github.com/openshift-metal3/dev-scripts/ci_token
   • CI_SERVER=api.ci.l2s4.p1.openshiftapps.com
   • CI_REGISTRY=quay-proxy.ci.openshift.org

2. Created a temporary kubeconfig and logged in with that token (same as write_pull_secret):

tmpkubeconfig=$(mktemp)
oc login "https://${CI_SERVER}:6443" --kubeconfig="$tmpkubeconfig" --token="${CI_TOKEN}"

Result: logged in as system:serviceaccount:openstack-k8s-operators:image-puller

3. Captured the username forms and token:

RAW_USER=$(oc --kubeconfig="$tmpkubeconfig" whoami)
SAFE_USER=$(oc --kubeconfig="$tmpkubeconfig" whoami | tr ':' '_')
TOKEN=$(oc --kubeconfig="$tmpkubeconfig" whoami -t)

RAW_USER=system:serviceaccount:openstack-k8s-operators:image-puller
SAFE_USER=system_serviceaccount_openstack-k8s-operators_image-puller

4. Reproduced the current master failure (raw whoami as username):

echo "$TOKEN" | podman login "$CI_REGISTRY" \
  --username "$RAW_USER" --password-stdin --authfile "$auth_bad"

Error: logging into "quay-proxy.ci.openshift.org": invalid username/password
rc=125

5. Applied this PR’s username transform and retried:

echo "$TOKEN" | podman login "$CI_REGISTRY" \
  --username "$SAFE_USER" --password-stdin --authfile "$auth_good"

Login Succeeded!
rc=0

Note

The local OCP kubeconfig under ocp/ocp/auth/kubeconfig is system:admin against api.ocp.openstack.lab — that is the deployed cluster and is not involved in this quay-proxy auth path. The proof above is against the build-farm SA, which is what the failing jobs hit.

Conclusion

Same token, same registry: colon-bearing SA username fails; colon-stripped username succeeds. That matches the intended fix in this PR.

[bfournie]

/lgtm

[openshift-ci]

@tusharjadhav3302: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-agent-compact-ipv4	08910ea	link	unknown	/test e2e-agent-compact-ipv4
ci/prow/e2e-metal-ipi-serial-ipv4-1of2	08910ea	link	unknown	/test e2e-metal-ipi-serial-ipv4-1of2
ci/prow/e2e-metal-ipi-serial-ipv4-2of2	08910ea	link	unknown	/test e2e-metal-ipi-serial-ipv4-2of2

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.