Add optional KMM integration for OoT kernel module loading by abyrne55 · Pull Request #79 · intel/gpu-base-operator

abyrne55 · 2026-08-05T20:00:15Z

This PR adds a KMMReconciler sub-controller that creates a KMM Module CR when the new ClusterPolicy.spec.kernelModule field is set and KMM is installed. The existing DP and DRA controllers keep their lifecycle logic. KMM handles only out-of-tree kernel module loading (modprobe, kernel mappings, in-cluster builds).

How it works

When ClusterPolicy.spec.kernelModule is set and KMM is installed in the cluster, KMMReconciler creates a KMM Module CR with a moduleLoader spec. The Module CR is owned by the ClusterPolicy and garbage-collected on deletion. A deletion guard skips Module CR removal while GPU ResourceClaims are still allocated.

KMM availability is detected at startup via API group discovery (kmm.sigs.x-k8s.io), same pattern as DRAEnable. If kernelModule is set but KMM isn't installed, the operator reports an error in ClusterPolicy status. If kernelModule is nil, the sub-controller is a no-op.

Downstream DaemonSets (DP, DRA, XPU Manager) are gated on the KMM ready node label (kmm.node.kubernetes.io/<ns>.<module>.ready), so they only schedule once the OoT module is actually loaded on a node. When kernelModule unset, however, their behavior is unchanged.

Relationship to PR #59

#59 proposed replacing the native DP/DRA controllers entirely with KMM. Two controllers collapse into one, and gpu-base-operator stops managing DP/DRA DaemonSets, RBAC, SCCs, and DeviceClasses at runtime. That gives the biggest code reduction (~-3,100 lines) but makes KMM a hard dependency for all users.

This PR takes a more conservative approach: the native controllers stay in place for DP/DRA lifecycle, and a new KMMReconciler runs alongside them solely for OoT kernel module management. KMM is never required, and clusters using in-tree drivers work exactly as before. The trade-off is more total code (+2,456 / -51 lines across 30 files).

`kernelModule` API

kernelMappings must contain at least 1 entry. Minimal usage: one OoT driver image matched to all kernels:

spec:
  kernelModule:
    kernelMappings:
    - regexp: "^.+$"
      containerImage: registry.example.com/xe-driver:1.0

More realistic usage: different images per kernel version

spec:
  kernelModule:
    moduleName: xe
    kernelMappings:
    - regexp: "^5\\.14\\.0-.*\\.el9.*\\.x86_64$"
      containerImage: registry.example.com/xe-rhel9:1.0
    - regexp: "^6\\.12\\..*"
      containerImage: registry.example.com/xe-rhel10:1.0

In-cluster build mode: useful when driver source is available but pre-built images don't exist for every kernel version in the fleet. KMM checks the target registry first and only triggers a build if the image is missing.

    kernelMappings:
    - regexp: "^5\\.14\\..*"
      build:
        dockerfileConfigMap:
          name: xe-dockerfile
        buildArgs:
        - name: XE_TAG
          value: v1.0
        secrets:
        - name: private-repo

The dockerfileConfigMap references a ConfigMap containing the Dockerfile. buildArgs are passed as build arguments, and secrets are mounted during the build (e.g., for private source repos). Registry auth should use pullSecret on ClusterPolicySpec instead.

Fields

Field	Description
`moduleName`	Kernel module to load. Defaults to `xe`.
`version`	Passed to the KMM container spec to trigger module redeploys on change.
`kernelMappings[].regexp`	Required. Kernel version pattern for this mapping.
`kernelMappings[].containerImage`	Pre-built OoT driver image for matching kernels.
`kernelMappings[].inTreeModulesToRemove`	Extra in-tree modules to unload for this mapping; `moduleName` is prepended automatically.
`inTreeModulesToRemove` (container level)	Auto-set to `[moduleName]` by the controller — the in-tree module is always unloaded before OoT insertion.
`modulesLoadingOrder`	Softdep-style loading order for multi-module drivers (first element must be `moduleName`, >=2 entries). Passed through as-is.
`firmwarePath`	In-container path for firmware files (maps to KMM `ModprobeSpec.FirmwarePath`).
`registryTLS`	Registry TLS options (`insecure`, `insecureSkipTLSVerify`) for the OoT image registry. Settable at top level and per mapping.

What's included

KernelModuleSpec / KernelMappingSpec / RegistryTLSSpec CRD types with DeepCopy
KMMReconciler sub-controller with CreateOrPatch, owner references, and deletion guard
KMM ready-label gating for downstream DP/DRA/XPU DaemonSets
Webhook validation and moduleName defaulting for kernelModule
OpenShift module-loader SCC, ServiceAccount, and RBAC via Helm templates
KMM Module RBAC (modules verbs) in the operator ClusterRole
Controller tests (~1,070 lines) and webhook validation tests (~635 lines) via envtest
Policy chart values and template for kernelModule configuration

Test results

Latest end-to-end run on OCP 4.22.9 SNO, 2× Intel BMG-G31 (Battlemage, PCI 8086:e223), kernel 5.14.0-687.35.1.el9_8.x86_64, KMM v2.7.0 + NFD pre-installed:

Scenario	Result
OoT + DRA (BMG-G31, in-tree driver replaced)	Module CR created with correct spec; KMM loaded OoT `xe` (`kmmStatus: 1/1`); ready label gated DP/DRA/XPU until load; DRA driver + XPU Manager `1/1`; ResourceSlice populated with both GPUs; DRA isolation test passed (`/dev/dri/` present in claimed container, absent otherwise); status errors cleared once healthy
CRD schema (simplified API)	New fields present, dropped fields absent; `moduleName` defaults to `xe`; `regexp` required (webhook-enforced)

Corroborating results from earlier runs (OCP 4.22.2, Arc Pro B70 with SR-IOV VFs):

Scenario	Result
In-tree, no KMM	DRA resources created, no Module CR, GPU workload OK, `kmmStatus: N/A`, clean deletion
OoT + DP	Module CR created, `gpu.intel.com/xe=8` registered, GPU workload OK

Known limitations

xe can't unload with active VFs. modprobe -rv cannot unload xe while SR-IOV VFs are active. KMM is adding modprobe.d support (kubernetes-sigs/kernel-module-management#1324) that could work around this in the future.

This PR was written in part with the assistance of generative AI.

pfl · 2026-08-10T09:50:55Z

+	// inserting the OOT module. ModuleName is always included implicitly.
+	// +optional
+	InTreeModulesToRemove []string `json:"inTreeModulesToRemove,omitempty"`
+


Is the 'InTree' prefix necessary here? Would there be cases in the future that the out-of-tree kernel module should be removed instead?

InTreeModulesToRemove matches the field name that KMM's Module CR uses. For each $MODULE_NAME passed in InTreeModulesToRemove, KMM checks if /lib/modules/.../$MODULE_NAME.ko exists (which is usually only true for non-KMM-installed modules, i.e., in-tree modules or modules installed via DKMS) before running modprobe -r $MODULE_NAME. KMM only removes KMM-installed OoT KMDs when their Module CR is deleted or when the node no longer matches the Module's node selector. One could argue that "PreExisting" or "OnNode" or "NonKMM" might be better at illustrating this nuance than "InTree", but doing so here would break upstream precedent and potentially confuse users.

So yes, I think the "InTree" prefix makes sense here. It matches established convention and signals to users that KMM-managed OoT modules shouldn't be listed here (they'd be silently skipped). I'm open to discussion though :)

pfl · 2026-08-10T11:04:59Z

+	// Literal is an exact kernel version string to match.
+	// Mutually exclusive with Regexp.
+	// +optional
+	Literal string `json:"literal,omitempty"`


Couldn't this be covered by Regexp above, which is just a string, e.g. "6.18.44"? Shouldn't we let the controller figure this out instead of having two fields for the same thing, especially since we have the "fallback" Image specified one step above?

It certainly can, and seeing as we're aiming to slim down the CRD, it shall 🙂. I'll drop Literal in favor of Regexp, and users can just do Regexp = "^6\\.18\\.44$" for exact-match situations

pfl · 2026-08-10T11:07:22Z

+	// When set, KMM builds the image if it doesn't exist in the registry.
+	// +optional
+	Build *KernelModuleBuildSpec `json:"build,omitempty"`
+


Once the system starts building, do we know if it succeeded and ready for use in order to proceed with any next steps dependent on the kernel module - or is this relevant for the use case?

We surface KMM's module loader status (available/desired) in ClusterPolicy.Status.KMMStatus for observability. For sequencing, I'll modify the DP/DRA/XPU Manager controllers to wait for KMM's kmm.node.kubernetes.io/{ns}.{name}.ready node label like @tkatila suggested. That label only appears once the module is successfully loaded, so pods won't schedule on nodes where the build/load hasn't completed.

pfl · 2026-08-10T11:09:12Z

+	// in order and unloads in reverse. Must have >=2 entries if set.
+	// +optional
+	ModulesLoadingOrder []string `json:"modulesLoadingOrder,omitempty"`
+


Do we repeat the 'xe' ModuleName in this array? Shouldn't this only contain dependencies for ModuleName?

KMM requires ModuleName as the first entry in this list, followed by its dependencies in order. I think it increases clarity to force users to explicitly declare the full ordering here (including xe/ModuleName), and that way we can just directly pass this field through to KMM. I'm open to discussion though :)

pfl · 2026-08-10T11:11:39Z

+
+	// FirmwarePath is the in-container path where firmware files are stored.
+	// +optional
+	FirmwarePath string `json:"firmwarePath,omitempty"`


There is no /usr/lib/firmware where to always look for the firmware(s)?

It depends on how the driver container image is built, and KMM (as we're just mirroring KMM's API here) doesn't set a default value here because an empty value means "no firmware to copy," which is the right default for modules that don't ship firmware

pfl · 2026-08-10T11:16:27Z

+	// for mappings that omit ContainerImage.
+	// +optional
+	Image string `json:"image,omitempty"`
+


Since we have ContainerImage in KerneMappings, should this be the DefaultImage instead? Or should KernelMappings be made mandatory with at least one entry instead, now there seems to be a bit of growth in information duplication. Whatever fine-grained information the kmm system needs, this operator can surely patch together?

Addressed below

tkatila

Some comments related to the CRD changes. I think the Module CR is somewhat complex as it has same fields in different places and would like to not copy them as they are. If possible.

tkatila · 2026-08-10T08:28:09Z

+type KernelModuleSpec struct {
+	// ModuleName is the kernel module to load (e.g., "xe").
+	// Also used as the default InTreeModulesToRemove entry.
+	ModuleName string `json:"moduleName"`


This should default to 'xe'. We don't have plans to support the OoT i915 driver. Variable could even be completely removed.

Noted! Will add the default value

tkatila · 2026-08-11T08:40:21Z

+	// SkipTLSVerify disables TLS certificate verification when pulling
+	// OOT driver images. For air-gapped or internal registries.
+	// +optional
+	SkipTLSVerify bool `json:"skipTLSVerify,omitempty"`


Let's use the registryTLS object from the Module instead. insecure and insecureSkipTLSVerify can be used in both container.registryTLS and container.kernelMappings.registryTLS

Agreed. I'll replace SkipTLSVerify with a RegistryTLS struct matching KMM's TLSOptions shape (insecure + insecureSkipTLSVerify) and allow it to be set both at the top level and per-mapping.

tkatila · 2026-08-11T08:43:08Z

 }

+// KernelModuleSpec configures out-of-tree kernel module loading via KMM.
+type KernelModuleSpec struct {


I think we need a version entry in the KernelModule. Xe OoT KMD has tagged versions and I think it could serve as a trigger for KMD updates?

Good call. KMM's ModuleLoaderContainerSpec has a Version field for exactly this purpose. Will add it to our struct and map it through

tkatila · 2026-08-11T08:46:37Z

+	// When KernelMappings is non-empty, serves as the KMM-level fallback
+	// for mappings that omit ContainerImage.
+	// +optional
+	Image string `json:"image,omitempty"`


Is using this same as having .* in the kernelMappings? If so, I think this can be dropped. KMDs are always tied to specific kernel versions, so it's not possible to have a generic container for many kernels.

It is indeed, and good point that it's not very useful for version-specific KMDs. I'll drop Image entirely and make KernelMappings required

tkatila · 2026-08-11T10:47:35Z

+	// Literal is an exact kernel version string to match.
+	// Mutually exclusive with Regexp.
+	// +optional
+	Literal string `json:"literal,omitempty"`


Needed if we just use Regexp?

Nope! Addressed above.

tkatila · 2026-08-11T11:36:22Z

+	"sigs.k8s.io/controller-runtime/pkg/client"
 )

 func generateNodeSelector(cp *v1alpha.ClusterPolicy) map[string]string {


Shouldn't the KMM's node labels be used if the KernelModule is enabled? i.e. only deploy Pods when the KMD is loaded?

Great idea. Addressed above.

tkatila · 2026-08-11T11:39:17Z

+	// InTreeModulesToRemove lists in-tree modules to unload before
+	// inserting the OOT module. ModuleName is always included implicitly.
+	// +optional
+	InTreeModulesToRemove []string `json:"inTreeModulesToRemove,omitempty"`


Can this be also removed as it's included in the kernelMappings object?

Sure, I'll drop the top-level InTreeModulesToRemove and keep the the per-mapping version (for cases where specific kernel versions need extra modules removed). I'll also make it such that gpu-base-operator always tells KMM to remove any existing in-tree modules named "xe" (or whatever ModuleName is set to), so users never need to specify it manually.

Add KMMReconciler sub-controller that creates a KMM Module CR when the new ClusterPolicy.spec.kernelModule field is set and KMM is installed. Native DP/DRA controllers are unchanged -- KMM is used only for out-of-tree kernel module management (modprobe, kernel mappings, in-cluster builds). Includes webhook validation, OpenShift SCC/RBAC support via Helm, and controller/webhook tests. Signed-off-by: Anthony Byrne <abyrne@redhat.com>

- Extract anyAllocatedResourceClaims to controller_utils.go so DRA and KMM share one implementation - Call updateStatus before the error return so KMMStatus reflects current module state even when CreateOrPatch fails - Deep-copy Secrets slice in convertBuildSpec to avoid sharing the backing array Signed-off-by: Anthony Byrne <abyrne@redhat.com>

Report module loader availability mismatches, stuck deletions, and ResourceClaim-blocked deletions in ClusterPolicy status errors. Signed-off-by: Anthony Byrne <abyrne@redhat.com>

tkatila · 2026-08-17T12:06:38Z

Thanks @abyrne55 for the changes! I'll try to use this on my end in the next couple of days.

Rework the KMM integration API based on review feedback: - Default moduleName to "xe" via CRD schema default and mutating webhook - Add version field mapping to KMM container.Version - Drop top-level image field; require kernelMappings (MinItems=1) - Replace skipTLSVerify bool with *RegistryTLSSpec (Insecure + InsecureSkipTLSVerify) - Drop literal field; require regexp on each kernel mapping - Gate downstream DaemonSets (DP, DRA, XPU) on KMM ready node label - Auto-set inTreeModulesToRemove from moduleName at container and per-mapping level - Pass modulesLoadingOrder through as-is - Clear stale Status.Errors each reconcile so it reflects current state Signed-off-by: Anthony Byrne <abyrne@redhat.com>

abyrne55 force-pushed the kmm-optional-v2 branch from 72f70ab to f6b068d Compare August 5, 2026 20:38

abyrne55 marked this pull request as ready for review August 6, 2026 16:19

abyrne55 requested review from pfl and tkatila as code owners August 6, 2026 16:19

pfl reviewed Aug 10, 2026

View reviewed changes

tkatila reviewed Aug 11, 2026

View reviewed changes

abyrne55 force-pushed the kmm-optional-v2 branch from 988d5fc to c2de30e Compare August 14, 2026 16:19

abyrne55 added 3 commits August 14, 2026 12:21

kmm: surface KMM status errors in ClusterPolicy

36d7f6a

Report module loader availability mismatches, stuck deletions, and ResourceClaim-blocked deletions in ClusterPolicy status errors. Signed-off-by: Anthony Byrne <abyrne@redhat.com>

abyrne55 force-pushed the kmm-optional-v2 branch from c2de30e to a85526a Compare August 14, 2026 16:23

abyrne55 force-pushed the kmm-optional-v2 branch from a85526a to 8e0e555 Compare August 17, 2026 17:30

Uh oh!

Conversation

abyrne55 commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How it works

kernelModule API

Fields

What's included

Test results

Known limitations

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abyrne55 Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abyrne55 Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abyrne55 Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tkatila left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abyrne55 Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abyrne55 Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tkatila commented Aug 17, 2026

Uh oh!

Reviewers

Assignees

abyrne55 commented Aug 5, 2026 •

edited

Loading

`kernelModule` API

abyrne55 Aug 13, 2026 •

edited

Loading

abyrne55 Aug 13, 2026 •

edited

Loading

abyrne55 Aug 13, 2026 •

edited

Loading

abyrne55 Aug 13, 2026 •

edited

Loading

abyrne55 Aug 13, 2026 •

edited

Loading