Skip to content

fix: preserve DCGM service compatibility #2918

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rahulait wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix: preserve DCGM service compatibility #2918

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions assets/state-dcgm-exporter/0800_daemonset.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ kind: DaemonSet
metadata:
labels:
app: nvidia-dcgm-exporter
nvidia.com/gpu-operator.dcgm-exporter: "true"

@tariq1890 tariq1890 Sep 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we look at alternate names for this label key?

The format of nvidia.com/gpu-operator.dcgm-exporter: "true" seems non-standard for labels.

@cdesiniotis cdesiniotis Sep 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe app.kubernetes.io/component: nvidia-dcgm-exporter?

@tariq1890 tariq1890 Sep 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that could work. It is a standardised label

@rahulait rahulait Sep 17, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, we can. This label was selected so that we use the same scheme we used before like here:

gpu-operator/internal/consts/consts.go

Line 71 in 93c0ff5

NVIDIADriverOwnerLabel = "nvidia.com/gpu-operator.driver.owner"

@tariq1890 tariq1890 Sep 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like app.kubernetes.io/component has a specific meaning as per this K8s docs

How about app.kubernetes.io/name instead?

@tariq1890 tariq1890 Sep 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rahulait I missed to mention this earlier, using a boolean as the label value typed seemed a bit off to me.

name: nvidia-dcgm-exporter
namespace: "FILLED BY THE OPERATOR"
annotations:
Expand All @@ -15,6 +16,7 @@ spec:
metadata:
labels:
app: nvidia-dcgm-exporter
nvidia.com/gpu-operator.dcgm-exporter: "true"
spec:
nodeSelector:
nvidia.com/gpu.deploy.dcgm-exporter: "true"
Expand Down
2 changes: 2 additions & 0 deletions assets/state-dcgm/0400_dcgm.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ kind: DaemonSet
metadata:
labels:
app: nvidia-dcgm
nvidia.com/gpu-operator.dcgm: "true"
name: nvidia-dcgm
namespace: "FILLED BY THE OPERATOR"
annotations:
Expand All @@ -15,6 +16,7 @@ spec:
metadata:
labels:
app: nvidia-dcgm
nvidia.com/gpu-operator.dcgm: "true"
spec:
nodeSelector:
nvidia.com/gpu.deploy.dcgm: "true"
Expand Down
22 changes: 22 additions & 0 deletions controllers/object_controls.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4740,6 +4740,7 @@ func DaemonSet(n ClusterPolicyController) (gpuv1.State, error) {
}

maps.Copy(obj.Labels, n.singleton.Spec.Daemonsets.Labels)
enforceDCGMCommonLabel(obj)

// Daemonsets will always have at least one annotation applied, so allocate if necessary
if obj.Annotations == nil {
Expand Down Expand Up @@ -4788,6 +4789,27 @@ func DaemonSet(n ClusterPolicyController) (gpuv1.State, error) {
return isDaemonSetReady(obj.Name, n), nil
}

func enforceDCGMCommonLabel(obj *appsv1.DaemonSet) {
var key string
switch obj.Name {
case "nvidia-dcgm", "nvidia-dcgm-dra":
key = "nvidia.com/gpu-operator.dcgm"
case "nvidia-dcgm-exporter", "nvidia-dcgm-exporter-dra":
key = "nvidia.com/gpu-operator.dcgm-exporter"
default:
return
}

if obj.Labels == nil {
obj.Labels = make(map[string]string)
}
obj.Labels[key] = "true"
if obj.Spec.Template.Labels == nil {
obj.Spec.Template.Labels = make(map[string]string)
}
obj.Spec.Template.Labels[key] = "true"
}
Comment thread
devin-ai-integration[bot] marked this conversation as resolved.

// isDaemonsetSpecChanged returns true if the spec has changed between existing one
// and new Daemonset spec compared by hash.
func isDaemonsetSpecChanged(current *appsv1.DaemonSet, new *appsv1.DaemonSet) bool {
Expand Down
76 changes: 76 additions & 0 deletions controllers/object_controls_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ const (
sandboxDevicePluginAssetsPath = "assets/state-sandbox-device-plugin"
kataDevicePluginAssetsPath = "assets/state-kata-device-plugin"
devicePluginAssetsPath = "assets/state-device-plugin/"
dcgmAssetsPath = "assets/state-dcgm/"
dcgmExporterAssetsPath = "assets/state-dcgm-exporter/"
migManagerAssetsPath = "assets/state-mig-manager/"
vGPUDeviceManagerAssetsPath = "assets/state-vgpu-device-manager/"
Expand Down Expand Up @@ -773,6 +774,24 @@ func testDaemonsetCommon(t *testing.T, cp *gpuv1.ClusterPolicy, component string
if err != nil {
return nil, fmt.Errorf("unable to get mainCtrImage for dcgm-exporter: %v", err)
}
case "DCGM":
spec = commonDaemonsetSpec{
repository: cp.Spec.DCGM.Repository,
image: cp.Spec.DCGM.Image,
version: cp.Spec.DCGM.Version,
imagePullPolicy: cp.Spec.DCGM.ImagePullPolicy,
imagePullSecrets: getImagePullSecrets(cp.Spec.DCGM.ImagePullSecrets),
args: cp.Spec.DCGM.Args,
env: cp.Spec.DCGM.Env,
resources: cp.Spec.DCGM.Resources,
}
dsLabel = "nvidia-dcgm"
mainCtrName = "nvidia-dcgm-ctr"
manifestFile = filepath.Join(cfg.root, dcgmAssetsPath)
mainCtrImage, err = gpuv1.ImagePath(&cp.Spec.DCGM)
if err != nil {
return nil, fmt.Errorf("unable to get mainCtrImage for dcgm: %v", err)
}
case "MIGManager":
spec = commonDaemonsetSpec{
repository: cp.Spec.MIGManager.Repository,
Expand Down Expand Up @@ -1681,6 +1700,63 @@ func getDCGMExporterTestOutput(testCase string) map[string]any {
return output
}

func TestDCGMCommonLabelEnforcement(t *testing.T) {
testCases := []struct {
description string
component string
label string
configure func(*gpuv1.ClusterPolicy)
}{
{
description: "DCGM",
component: "DCGM",
label: "nvidia.com/gpu-operator.dcgm",
configure: func(cp *gpuv1.ClusterPolicy) {
enabled := true
cp.Spec.DCGM.Enabled = &enabled
cp.Spec.DCGM.Repository = "nvcr.io/nvidia/cloud-native"
cp.Spec.DCGM.Image = "dcgm"
cp.Spec.DCGM.Version = "3.3.0-1-ubuntu22.04"
},
},
{
description: "DCGMExporter",
component: "DCGMExporter",
label: "nvidia.com/gpu-operator.dcgm-exporter",
configure: func(cp *gpuv1.ClusterPolicy) {
cp.Spec.DCGMExporter.Repository = "nvcr.io/nvidia/k8s"
cp.Spec.DCGMExporter.Image = "dcgm-exporter"
cp.Spec.DCGMExporter.Version = "3.3.0-3.2.0-ubuntu22.04"
},
},
}

for _, tc := range testCases {
t.Run(tc.description, func(t *testing.T) {
cp := clusterPolicy.DeepCopy()
cp.Spec.Validator.Repository = "nvcr.io/nvidia/cloud-native"
cp.Spec.Validator.Image = "gpu-operator-validator"
cp.Spec.Validator.Version = "v23.9.2"
tc.configure(cp)
cp.Spec.Daemonsets.Labels = map[string]string{
tc.label: "false",
"custom": "label",
}

ds, err := testDaemonsetCommon(t, cp, tc.component, 1)
require.NoError(t, err)
require.NotNil(t, ds)
require.Equal(t, "true", ds.Labels[tc.label])
require.Equal(t, "true", ds.Spec.Template.Labels[tc.label])
require.Equal(t, "label", ds.Labels["custom"])
require.Equal(t, "label", ds.Spec.Template.Labels["custom"])

require.NoError(t, removeState(&clusterPolicyController, clusterPolicyController.idx-1))
clusterPolicyController.idx--
})
}
}

// TestDCGMExporter tests that the GPU Operator correctly deploys the dcgm-exporter daemonset
// under various scenarios/config options
func TestDCGMExporter(t *testing.T) {
Expand Down
37 changes: 30 additions & 7 deletions internal/state/dcgm_exporter_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ func TestDCGMExporterEnabledByDefault(t *testing.T) {
assert.Equal(t, 1, kinds["RoleBinding"])
assert.Equal(t, 1, kinds["ResourceClaimTemplate"])
assert.Equal(t, 1, kinds["DaemonSet"])
assert.Equal(t, 1, kinds["Service"])
assert.Equal(t, 2, kinds["Service"])
// DRA attribution needs the ResourceSlice informer, so the read ClusterRole is
// always bound. No ServiceMonitor by default.
assert.Equal(t, 1, kinds["ClusterRole"])
Expand All @@ -89,6 +89,8 @@ func TestDCGMExporterEnabledByDefault(t *testing.T) {
claimHasAdminAccess(t, findByKind(objs, "ResourceClaimTemplate"))

ds := findDaemonSet(t, objs)
assert.Equal(t, "true", ds.Labels["nvidia.com/gpu-operator.dcgm-exporter"])
assert.Equal(t, "true", ds.Spec.Template.Labels["nvidia.com/gpu-operator.dcgm-exporter"])
podSpec := ds.Spec.Template.Spec
assert.Equal(t, "true", podSpec.NodeSelector["nvidia.com/gpu.deploy.dcgm-exporter-dra"])
require.NotNil(t, podSpec.AutomountServiceAccountToken)
Expand Down Expand Up @@ -228,10 +230,31 @@ func TestDCGMExporterServiceType(t *testing.T) {
objs, err := s.getManifestObjects(context.Background(), cr, draSupportedCatalog())
require.NoError(t, err)

svc := findByKind(objs, "Service")
require.NotNil(t, svc)
svcType, _, _ := unstructured.NestedString(svc.Object, "spec", "type")
assert.Equal(t, "NodePort", svcType)
itpValue, _, _ := unstructured.NestedString(svc.Object, "spec", "internalTrafficPolicy")
assert.Equal(t, "Local", itpValue)
for _, name := range []string{"nvidia-dcgm-exporter", "nvidia-dcgm-exporter-dra"} {
svc := findByKindAndName(objs, "Service", name)
require.NotNil(t, svc)
svcType, _, _ := unstructured.NestedString(svc.Object, "spec", "type")
assert.Equal(t, "NodePort", svcType)
itpValue, _, _ := unstructured.NestedString(svc.Object, "spec", "internalTrafficPolicy")
assert.Equal(t, "Local", itpValue)
selector, _, _ := unstructured.NestedStringMap(svc.Object, "spec", "selector")
assert.Equal(t, map[string]string{"app": "nvidia-dcgm-exporter-dra"}, selector)
if name == "nvidia-dcgm-exporter" {
assert.Equal(t, "true", svc.GetAnnotations()["prometheus.io/scrape"])
} else {
assert.NotContains(t, svc.GetAnnotations(), "prometheus.io/scrape")
}
}
}

func TestDCGMExporterCommonLabelCannotBeOverridden(t *testing.T) {
s := newTestDCGMExporterState(t, false)
cr := exporterCR(&nvidiav1.DCGMExporterSpec{})
cr.Spec.Daemonsets.Labels = map[string]string{"nvidia.com/gpu-operator.dcgm-exporter": "false"}

objs, err := s.getManifestObjects(context.Background(), cr, draSupportedCatalog())
require.NoError(t, err)
ds := findDaemonSet(t, objs)
assert.Equal(t, "true", ds.Labels["nvidia.com/gpu-operator.dcgm-exporter"])
assert.Equal(t, "true", ds.Spec.Template.Labels["nvidia.com/gpu-operator.dcgm-exporter"])
}
44 changes: 37 additions & 7 deletions internal/state/dcgm_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,15 @@ func findByKind(objs []*unstructured.Unstructured, kind string) *unstructured.Un
return nil
}

func findByKindAndName(objs []*unstructured.Unstructured, kind, name string) *unstructured.Unstructured {
for _, o := range objs {
if o.GetKind() == kind && o.GetName() == name {
return o
}
}
return nil
}

// claimHasAdminAccess verifies the ResourceClaimTemplate requests all GPUs with adminAccess.
func claimHasAdminAccess(t *testing.T, rct *unstructured.Unstructured) {
t.Helper()
Expand Down Expand Up @@ -117,11 +126,13 @@ func TestDCGMEnabled(t *testing.T) {
assert.Equal(t, 1, kinds["RoleBinding"])
assert.Equal(t, 1, kinds["ResourceClaimTemplate"])
assert.Equal(t, 1, kinds["DaemonSet"])
assert.Equal(t, 1, kinds["Service"])
assert.Equal(t, 2, kinds["Service"])

claimHasAdminAccess(t, findByKind(objs, "ResourceClaimTemplate"))

ds := findDaemonSet(t, objs)
assert.Equal(t, "true", ds.Labels["nvidia.com/gpu-operator.dcgm"])
assert.Equal(t, "true", ds.Spec.Template.Labels["nvidia.com/gpu-operator.dcgm"])
podSpec := ds.Spec.Template.Spec
assert.Equal(t, "true", podSpec.NodeSelector["nvidia.com/gpu.deploy.dcgm-dra"])
require.Len(t, podSpec.Containers, 1)
Expand All @@ -138,13 +149,32 @@ func TestDCGMEnabled(t *testing.T) {
require.Len(t, ctr.Resources.Claims, 1)
assert.Equal(t, "admin-gpus", ctr.Resources.Claims[0].Name)

// The exporter targets this Service on port 5555.
svc := findByKind(objs, "Service")
port, found, err := unstructured.NestedSlice(svc.Object, "spec", "ports")
for _, name := range []string{"nvidia-dcgm", "nvidia-dcgm-dra"} {
svc := findByKindAndName(objs, "Service", name)
require.NotNil(t, svc)
selector, found, err := unstructured.NestedStringMap(svc.Object, "spec", "selector")
require.NoError(t, err)
require.True(t, found)
assert.Equal(t, map[string]string{"app": "nvidia-dcgm-dra"}, selector)
port, found, err := unstructured.NestedSlice(svc.Object, "spec", "ports")
require.NoError(t, err)
require.True(t, found)
require.Len(t, port, 1)
assert.Equal(t, int64(5555), port[0].(map[string]any)["port"])
}
}

func TestDCGMCommonLabelCannotBeOverridden(t *testing.T) {
s := newTestDCGMState(t)
cr := sampleGPUCluster()
cr.Spec.DCGM = &nvidiav1.DCGMSpec{Enabled: new(true)}
cr.Spec.Daemonsets.Labels = map[string]string{"nvidia.com/gpu-operator.dcgm": "false"}

objs, err := s.getManifestObjects(context.Background(), cr, draSupportedCatalog())
require.NoError(t, err)
require.True(t, found)
require.Len(t, port, 1)
assert.Equal(t, int64(5555), port[0].(map[string]any)["port"])
ds := findDaemonSet(t, objs)
assert.Equal(t, "true", ds.Labels["nvidia.com/gpu-operator.dcgm"])
assert.Equal(t, "true", ds.Spec.Template.Labels["nvidia.com/gpu-operator.dcgm"])
}

func TestDCGMImageFromEnvFallback(t *testing.T) {
Expand Down
78 changes: 78 additions & 0 deletions internal/state/state_skel.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import (

"github.com/go-logr/logr"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/api/meta"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
Expand Down Expand Up @@ -442,9 +443,86 @@ func (s *stateSkel) mergeObjects(updated, current *unstructured.Unstructured) er
if gvk.Group == "" && gvk.Kind == "ServiceAccount" {
return s.mergeServiceAccount(updated, current)
}
if gvk.Group == "" && gvk.Kind == "Service" {
return s.mergeService(updated, current)
}
return nil
}

// mergeService preserves fields allocated by the API server. Clearing these fields in a full
// update is rejected because their values are immutable or must remain allocated.
func (s *stateSkel) mergeService(updated, current *unstructured.Unstructured) error {

@cdesiniotis cdesiniotis Sep 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question -- why is this needed now when it wasn't needed before? Is it because this commit is introducing a diff in the nvidia-dcgm-exporter-dra Service object?

@rahulait rahulait Sep 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was flagged by LLMs. I believe we had this bug introduced recently but never caught it.

non-dra path uses:

gpu-operator/controllers/object_controls.go

Lines 4952 to 4956 in 93c0ff5

logger.Info("Found Resource, updating...")
obj.ResourceVersion = found.ResourceVersion
obj.Spec.ClusterIP = found.Spec.ClusterIP
err = n.client.Update(ctx, obj)

It sets resourceVersion and clusterip allocated before calling update().

However, GPUCluster path uses generic unstructured reconciler which only preserves resourceVersion but not the clusterIP. Kubernetes can reject the update if clusterIP is omitted. This might be an existing bug in v26.7.0 not yet reported. Once service is created and ip is assigned, future reconcile updates might be failing but its not getting affected as it exists, just not getting updated.

updatedService := &corev1.Service{}
if err := runtime.DefaultUnstructuredConverter.FromUnstructured(updated.Object, updatedService); err != nil {
return fmt.Errorf("failed to convert updated Service: %w", err)
}
currentService := &corev1.Service{}
if err := runtime.DefaultUnstructuredConverter.FromUnstructured(current.Object, currentService); err != nil {
return fmt.Errorf("failed to convert current Service: %w", err)
}

if updatedService.Spec.ClusterIP == "" && len(updatedService.Spec.ClusterIPs) == 0 {
updatedService.Spec.ClusterIP = currentService.Spec.ClusterIP
updatedService.Spec.ClusterIPs = currentService.Spec.ClusterIPs
}
if len(updatedService.Spec.IPFamilies) == 0 {
updatedService.Spec.IPFamilies = currentService.Spec.IPFamilies
}
if updatedService.Spec.IPFamilyPolicy == nil {
updatedService.Spec.IPFamilyPolicy = currentService.Spec.IPFamilyPolicy
}

if serviceAllocatesNodePorts(updatedService) {
for i := range updatedService.Spec.Ports {
if updatedService.Spec.Ports[i].NodePort != 0 {
continue
}
if currentPort := findMatchingServicePort(updatedService.Spec.Ports[i], currentService.Spec.Ports); currentPort != nil {
updatedService.Spec.Ports[i].NodePort = currentPort.NodePort
}
}
}
if updatedService.Spec.Type == corev1.ServiceTypeLoadBalancer &&
updatedService.Spec.ExternalTrafficPolicy == corev1.ServiceExternalTrafficPolicyLocal &&
updatedService.Spec.HealthCheckNodePort == 0 {
updatedService.Spec.HealthCheckNodePort = currentService.Spec.HealthCheckNodePort
}

merged, err := runtime.DefaultUnstructuredConverter.ToUnstructured(updatedService)
if err != nil {
return fmt.Errorf("failed to convert merged Service: %w", err)
}
updated.Object = merged
return nil
}

func serviceAllocatesNodePorts(service *corev1.Service) bool {
if service.Spec.Type == corev1.ServiceTypeNodePort {
return true
}
return service.Spec.Type == corev1.ServiceTypeLoadBalancer &&
(service.Spec.AllocateLoadBalancerNodePorts == nil || *service.Spec.AllocateLoadBalancerNodePorts)
}

func findMatchingServicePort(updated corev1.ServicePort, current []corev1.ServicePort) *corev1.ServicePort {
for i := range current {
if updated.Name != "" && updated.Name == current[i].Name {
return &current[i]
}
if updated.Name == "" && current[i].Name == "" && servicePortProtocol(updated) == servicePortProtocol(current[i]) {
return &current[i]
}
}
return nil
}

func servicePortProtocol(port corev1.ServicePort) corev1.Protocol {
if port.Protocol == "" {
return corev1.ProtocolTCP
}
return port.Protocol
}

// For Service Account, keep secrets if exists
func (s *stateSkel) mergeServiceAccount(updated, current *unstructured.Unstructured) error {
curSecrets, ok, err := unstructured.NestedSlice(current.Object, "secrets")
Comment on lines 443 to 528

@coderabbitai coderabbitai Bot Sep 17, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🔎 Supported by static analysis

🏁 Script executed:

sed -n '430,540p' internal/state/state_skel.go
sed -n '1,100p' manifests/state-dcgm/0550_service.yaml
sed -n '1,120p' manifests/state-dcgm-exporter/0490_service.yaml
sed -n '4910,5000p' controllers/object_controls.go
rg -n 'nvidia-dcgm(-exporter)?|selector|mergeObjects|mergeService' internal/state controllers manifests assets | head -240

Repository: NVIDIA/gpu-operator

Length of output: 31109

🏁 Script executed:

#!/bin/bash
set -e
printf '%s\n' '--- state skeleton reconciliation ---'
sed -n '280,390p' internal/state/state_skel.go
printf '%s\n' '--- state skeleton declarations and callers ---'
rg -n -C 3 'NewState|stateSkel|Reconcile|mergeObjects|GPUCluster|ClusterPolicy|State\(' internal/state controllers | head -320
printf '%s\n' '--- DCGM state definitions and registration ---'
rg -n -C 5 'func (.*DCGM|func DCGM|DCGMExporter|dcgm|state-dcgm|state-dcgm-exporter' internal/state controllers | head -360
printf '%s\n' '--- Service update paths and selector preservation ---'
rg -n -C 5 'Spec\.Selector|Selector.*current|current.*Selector|preserve.*selector|selector.*preserve|Service\(' internal/state controllers | head -360
printf '%s\n' '--- rollout/upgrade requirement evidence ---'
rg -n -i -C 4 'selector.*(preserv|rollout|upgrade)|preserv.*selector|endpoint|downtime|availability|DCGM' . --glob '!internal/state/testdata/golden/**' --glob '!vendor/**' | head -360

Repository: NVIDIA/gpu-operator

Length of output: 50375

🤖 get_repo_knowledge executed:

get_repo_knowledge NVIDIA/gpu-operator /tmp/coderabbit-repo-knowledge/nvidia-gpu-operator-eed98df4/conventions

Length of output: 4401

🏁 Script executed:

set -e
sed -n '280,390p' internal/state/state_skel.go
rg -n -C 4 'stateSkel|mergeObjects|GPUCluster|state-dcgm|state-dcgm-exporter|Spec\.Selector|preserve.*selector|selector.*preserv' internal/state controllers

Repository: NVIDIA/gpu-operator

Length of output: 50376

🏁 Script executed:

set -e
printf '%s\n' '--- changed state skeleton diff ---'
git diff -- internal/state/state_skel.go
printf '%s\n' '--- state manager/controller GPUCluster entrypoints ---'
rg -n -C 8 'GPUCluster|SyncState|Sync\(|NewStateDCGM|NewStateDCGMExporter|newStates|stateManager|Reconcile' controllers/state_manager.go internal/state/manager.go internal/state/*.go | head -500
printf '%s\n' '--- focused selector tests and requirements ---'
rg -n -C 8 'selector|endpoint|rollout|preserv|DRA' internal/state/state_skel*_test.go internal/state/dcgm*_test.go internal/state/*test.go controllers/*test.go docs README.md CONTRIBUTING.md 2>/dev/null | head -500

Repository: NVIDIA/gpu-operator

Length of output: 50375

🏁 Script executed:

set -e
printf '%s\n' '--- internal state manager ---'
cat -n internal/state/manager.go
printf '%s\n' '--- direct state manager invocations ---'
rg -n -C 10 'SyncState\(|\.Sync\(ctx|Sync\(.*GPUCluster|SyncState\(' controllers internal --glob '*.go' | head -320
printf '%s\n' '--- state factory registration ---'
rg -n -C 12 'NewStates|NewStateDCGM|NewStateDCGMExporter|GPUClusterCRDName|GPUCluster' internal/state/manager.go controllers | head -320

Repository: NVIDIA/gpu-operator

Length of output: 50375

Preserve existing Service selectors during GPUCluster updates. GPUClusterReconciler reaches the DCGM and DCGM exporter states, and createOrUpdateObjs updates existing Services through mergeService. That helper preserves allocated fields but leaves the desired Spec.Selector unchanged. An upgrade can therefore replace a selector for serving legacy pods with app: nvidia-dcgm-dra or app: nvidia-dcgm-exporter-dra before those pods are ready. Kubernetes then removes the legacy pods from the Service endpoints, causing a temporary hostengine or metrics outage.

Copy the current selector in mergeService for existing Services. Keep the DRA selector for newly created Services.

Expand Down
Loading
Loading