Skip to content

[Feature]: Allow overriding the derived OS-tag suffix on driver image versions - OKD/SCOS nodes (os_release ID=centos) resolve to nonexistent -centos10 tags #2892

Description

@aqeelat

In OKD deployments (okd-project/okd) the nodes run CentOS Stream CoreOS (SCOS), whose NFD labels report feature.node.kubernetes.io/system-os_release.ID=centos and …VERSION_ID=10. getOSTag() in internal/state/nodepool.go therefore renders the driver image suffix as centos10 → nvcr.io/nvidia/driver:<version>-centos10, a tag that is not published, and the driver pod fails with ImagePullBackOff. RHEL and its clones are already special-cased there (ol, rocky, rhel → major-only suffix); centos falls through to the default branch.

The documented workaround (confirmed in #1533) is pinning spec.driver.version to a digest, which is used verbatim. That works, but freezes the image generation: rebuilds never flow under the same pin, leaving no tag-based upgrade path for the pinned reference. In practice we also hit the pinned tag being removed from nvcr.io (gpu-driver-container#985), and the newer images we would otherwise move to were blocked by a separate issue (gpu-driver-container#984).

Expected behavior: users running OKD/SCOS should be able to select a published, compatible driver image suffix (or otherwise prevent the operator from constructing an unavailable -centos10 tag).

One shape that would solve it: a CRD-level override of the derived OS tag — e.g. spec.driver.osTag: rhel10.0 — taking precedence over the os_release-derived suffix. A smaller alternative in the same spirit: extending getOSTag's existing ol/rocky/rhel normalization to map centos → rhel. For context on compatibility: OKD is the community OpenShift distribution and the operator's OpenShift/DTK integration is active on these nodes (not a standalone CentOS host), and we build against the published -rhel10 driver images on SCOS today (digest-pinned 580.126.20-rhel10.0, kernel 6.12.0-233.el10) with the kmod building and loading fine. Happy with either shape — whichever better fits the project's support model.

Related: #1533 (original ask, resolved via the digest workaround, closed stale) · #2811 (downstream: repoConfig in DTK container) · gpu-driver-container#984 / #985

Environment: GPU Operator v26.3.3 · OKD 4.22.0-okd-scos.6 · CentOS Stream CoreOS 10, kernel 6.12.0-233.el10.x86_64 · NVIDIA L4 · amd64

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions