Skip to content

[Bug]: GPU Operator MPS config-manager cannot signal MPS daemon due to process-target mismatch #2970

Open
Open
[Bug]: GPU Operator MPS config-manager cannot signal MPS daemon due to process-target mismatch#2970
Labels
bugIssue/PR to expose/discuss/fix a bugneeds-triageissue or PR has not been assigned a priority-px label

Description

@anxolerd
anxolerd
opened on Sep 25, 2026

Describe the bug
With GPU Operator v26.7.0 and v26.7.1 (other versions not tested), the MPS control-daemon DaemonSet config-manager sidecar cannot reload the MPS control daemon after a device-plugin configuration change.

The MPS DaemonSet template starts the daemon using: command: [mps-control-daemon] but config-manager is configured to search for: PROCESS_TO_SIGNAL: /usr/bin/mps-control-daemon. The config-manager implementation requires an exact match against cmdline[0]. It cannot find the process, exits with code 1, and enters CrashLoopBackOff.

This leaves the MPS DaemonSet at 1/2 ready and the GPU Operator ClusterPolicy in notReady, even when the MPS main container is eventually serving shared-GPU resources.

To Reproduce

  1. Install GPU Operator v26.7.1 with a device-plugin ConfigMap containing two profiles: i.e. default without MPS (full GPU allocation) and one with MPS enabled. The values I used:
mig:
  strategy: none

devicePlugin:
  config:
    create: true
    name: nvidia-device-plugin-config
    default: fullGpu
    data:
      fullGpu: |-
        version: v1
        flags:
          migStrategy: none
      mps-profile: |-
        version: v1
        sharing:
          mps:
            renameByDefault: true
            resources:
              - name: nvidia.com/gpu
                replicas: 4
  mps:
    root: /run/nvidia/mps
  1. Allow daemons to start with default profile:
  2. Change the node to the MPS profile:
kubectl label node <gpu-node> nvidia.com/device-plugin.config=mps-profile --overwrite
  1. Check the MPS config-manager logs:
POD=$(kubectl get pods -n nvidia -l app=nvidia-device-plugin-mps-control-daemon -o jsonpath='{.items[0].metadata.name}')
kubectl logs -n nvidia "$POD" -c config-manager --tail=100

Actual result

Label change detected: nvidia.com/device-plugin.config=mps-profile
Updating to config: mps-profile
Successfully updated to config: mps-profile
Sending signal 'hangup' to '/usr/bin/mps-control-daemon'
error finding pid: no process found

The config-manager sidecar enters CrashLoopBackOff, the MPS DaemonSet remains 1/2, and ClusterPolicy stays notReady.

Expected behavior
The config-manager sends SIGHUP successfully, the MPS control daemon reloads the new profile, and the MPS DaemonSet remains 2/2 Ready.

Environment (please provide the following information):

  • GPU Operator Version: v26.7.1
  • OS: Ubuntu24.04
  • Kernel Version: 6.8.0-generic
  • Container Runtime Version: containerd 2.2.2-1ubuntu.24.04noble
  • Kubernetes Distro and Version: Kubernetes: v1.35.3

Information to attach (optional if deemed irrelevant)

  • kubernetes pods status: kubectl get pods -n OPERATOR_NAMESPACE

    # Other pods are running and completely healthy except the mps-control-daemon
...
nvidia-device-plugin-mps-control-daemon-wqhfq                     1/2     CrashLoopBackOff   12 (106s ago)   39m
...

  • kubernetes daemonset status: kubectl get ds -n OPERATOR_NAMESPACE

    NAME                                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                                                          AGE
nvidia-device-plugin-mps-control-daemon   1         1         0       1            0           nvidia.com/gpu.deploy.device-plugin=true,nvidia.com/mps.capable=true   3h36m

  • If a pod/ds is in an error state or pending state kubectl describe pod -n OPERATOR_NAMESPACE POD_NAME

     $ kubectl describe pod -n nvidia nvidia-device-plugin-mps-control-daemon-8vqms
 Name:                 nvidia-device-plugin-mps-control-daemon-8vqms
 Namespace:            nvidia
 Priority:             2000001000
 Priority Class Name:  system-node-critical
 Runtime Class Name:   nvidia
 Service Account:      nvidia-device-plugin
 Node:                 k8s-worker-gpu-0/172.22.4.22
 Start Time:           Fri, 25 Sep 2026 19:12:17 +0300
 Labels:               app=nvidia-device-plugin-mps-control-daemon
                       app.kubernetes.io/managed-by=gpu-operator
                       controller-revision-hash=956ddf497
                       helm.sh/chart=gpu-operator-v26.7.1
                       pod-template-generation=8
                       topology.kubernetes.io/region=RegionOne
                       topology.kubernetes.io/zone=nova
 Annotations:          <none>
 Status:               Running
 IP:                   10.244.221.17
 IPs:
   IP:           10.244.221.17
 Controlled By:  DaemonSet/nvidia-device-plugin-mps-control-daemon
 Init Containers:
   toolkit-validation:
     Container ID:  containerd://828c8ff9e644722a867b3df8d520a65dc64dea80a1be2770f9cc8419e4d3fa5c
     Image:         nvcr.io/nvidia/gpu-operator:v26.7.1
     Image ID:      nvcr.io/nvidia/gpu-operator@sha256:b844f9c033e748b8f9925f791f426a531ee8a855f04d4ccea365378634bcefe5
     Port:          <none>
     Host Port:     <none>
     Command:
       sh
       -c
     Args:
       until [ -f /run/nvidia/validations/toolkit-ready ] && { grep -q '^nvidia ' /proc/modules || [ -e /dev/dxg ]; }; do echo waiting for nvidia container stack to be setup; sleep 5; done
     State:          Terminated
       Reason:       Completed
       Exit Code:    0
       Started:      Fri, 25 Sep 2026 19:12:17 +0300
       Finished:     Fri, 25 Sep 2026 19:12:17 +0300
     Ready:          True
     Restart Count:  0
     Environment:    <none>
     Mounts:
       /run/nvidia from run-nvidia (rw)
       /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z6xkw (ro)
   mps-control-daemon-mounts:
     Container ID:  containerd://6b72cb1e4350fd323d4a85bb8522412252f321be8868c488b923d5e4663c405b
     Image:         nvcr.io/nvidia/k8s-device-plugin:v0.20.1
     Image ID:      nvcr.io/nvidia/k8s-device-plugin@sha256:27c1b2553a690ca3d29889dcd61df97525d46f58272c0be8dc63c2b18f120075
     Port:          <none>
     Host Port:     <none>
     Command:
       mps-control-daemon
       mount-shm
     State:          Terminated
       Reason:       Completed
       Exit Code:    0
       Started:      Fri, 25 Sep 2026 19:12:18 +0300
       Finished:     Fri, 25 Sep 2026 19:12:18 +0300
     Ready:          True
     Restart Count:  0
     Environment:    <none>
     Mounts:
       /mps from mps-root (rw)
       /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z6xkw (ro)
   config-manager-init:
     Container ID:  containerd://e81e9d6ce3c5fca148f959fa6558c661618d4520e301270f373f2314f40d5e82
     Image:         nvcr.io/nvidia/k8s-device-plugin:v0.20.1
     Image ID:      nvcr.io/nvidia/k8s-device-plugin@sha256:27c1b2553a690ca3d29889dcd61df97525d46f58272c0be8dc63c2b18f120075
     Port:          <none>
     Host Port:     <none>
     Command:
       config-manager
     State:          Terminated
       Reason:       Completed
       Exit Code:    0
       Started:      Fri, 25 Sep 2026 19:12:19 +0300
       Finished:     Fri, 25 Sep 2026 19:12:19 +0300
     Ready:          True
     Restart Count:  0
     Environment:
       ONESHOT:              true
       KUBECONFIG:
       NODE_NAME:             (v1:spec.nodeName)
       NODE_LABEL:           nvidia.com/device-plugin.config
       CONFIG_FILE_SRCDIR:   /available-configs
       CONFIG_FILE_DST:      /config/config.yaml
       DEFAULT_CONFIG:       fullGpu
       SEND_SIGNAL:          false
       SIGNAL:
       PROCESS_TO_SIGNAL:
       FALLBACK_STRATEGIES:  empty
     Mounts:
       /available-configs from nvidia-gpu-operator-device-plugin-config (rw)
       /config from config (rw)
       /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z6xkw (ro)
 Containers:
   mps-control-daemon-ctr:
     Container ID:  containerd://7fe7a1e57c4cc978869e6da1cff72439c0241c236c3826550fb6b816d0409c20
     Image:         nvcr.io/nvidia/k8s-device-plugin:v0.20.1
     Image ID:      nvcr.io/nvidia/k8s-device-plugin@sha256:27c1b2553a690ca3d29889dcd61df97525d46f58272c0be8dc63c2b18f120075
     Port:          <none>
     Host Port:     <none>
     Command:
       mps-control-daemon
     State:          Running
       Started:      Fri, 25 Sep 2026 19:12:20 +0300
     Ready:          True
     Restart Count:  0
     Environment:
       NODE_NAME:                    (v1:spec.nodeName)
       NVIDIA_VISIBLE_DEVICES:      all
       NVIDIA_DRIVER_CAPABILITIES:  compute,utility
       CONFIG_FILE:                 /config/config.yaml
       MIG_STRATEGY:                none
     Mounts:
       /available-configs from nvidia-gpu-operator-device-plugin-config (rw)
       /config from config (rw)
       /dev/shm from mps-shm (rw)
       /mps from mps-root (rw)
       /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z6xkw (ro)
   config-manager:
     Container ID:  containerd://83d50e253dd1b7c1393fe4188b6f20fc9f82786190234ee6128fb9ec25586141
     Image:         nvcr.io/nvidia/k8s-device-plugin:v0.20.1
     Image ID:      nvcr.io/nvidia/k8s-device-plugin@sha256:27c1b2553a690ca3d29889dcd61df97525d46f58272c0be8dc63c2b18f120075
     Port:          <none>
     Host Port:     <none>
     Command:
       config-manager
     State:          Terminated
       Reason:       Error
       Exit Code:    1
       Started:      Fri, 25 Sep 2026 19:12:34 +0300
       Finished:     Fri, 25 Sep 2026 19:12:34 +0300
     Last State:     Terminated
       Reason:       Error
       Exit Code:    1
       Started:      Fri, 25 Sep 2026 19:12:21 +0300
       Finished:     Fri, 25 Sep 2026 19:12:21 +0300
     Ready:          False
     Restart Count:  2
     Environment:
       ONESHOT:              false
       KUBECONFIG:
       NODE_NAME:             (v1:spec.nodeName)
       NODE_LABEL:           nvidia.com/device-plugin.config
       CONFIG_FILE_SRCDIR:   /available-configs
       CONFIG_FILE_DST:      /config/config.yaml
       DEFAULT_CONFIG:       fullGpu
       SEND_SIGNAL:          true
       SIGNAL:               1
       PROCESS_TO_SIGNAL:    /usr/bin/mps-control-daemon
       FALLBACK_STRATEGIES:  empty
     Mounts:
       /available-configs from nvidia-gpu-operator-device-plugin-config (rw)
       /config from config (rw)
       /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z6xkw (ro)
 Conditions:
   Type                        Status
   PodReadyToStartContainers   True
   Initialized                 True
   Ready                       False
   ContainersReady             False
   PodScheduled                True
 Volumes:
   run-nvidia:
     Type:          HostPath (bare host directory volume)
     Path:          /run/nvidia
     HostPathType:  Directory
   mps-root:
     Type:          HostPath (bare host directory volume)
     Path:          /run/nvidia/mps
     HostPathType:  DirectoryOrCreate
   mps-shm:
     Type:          HostPath (bare host directory volume)
     Path:          /run/nvidia/mps/shm
     HostPathType:
   nvidia-gpu-operator-device-plugin-config:
     Type:      ConfigMap (a volume populated by a ConfigMap)
     Name:      nvidia-gpu-operator-device-plugin-config
     Optional:  false
   config:
     Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
     Medium:
     SizeLimit:  <unset>
   kube-api-access-z6xkw:
     Type:                    Projected (a volume that contains injected data from multiple sources)
     TokenExpirationSeconds:  3607
     ConfigMapName:           kube-root-ca.crt
     Optional:                false
     DownwardAPI:             true
 QoS Class:                   BestEffort
 Node-Selectors:              nvidia.com/gpu.deploy.device-plugin=true
                              nvidia.com/mps.capable=true
 Tolerations:                 evo.company/nvidia-gpu:NoSchedule op=Exists
                              node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                              node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                              node.kubernetes.io/not-ready:NoExecute op=Exists
                              node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                              node.kubernetes.io/unreachable:NoExecute op=Exists
                              node.kubernetes.io/unschedulable:NoSchedule op=Exists
                              nvidia.com/gpu:NoSchedule op=Exists
 Events:
   Type     Reason     Age                From               Message
   ----     ------     ----               ----               -------
   Normal   Scheduled  43s                default-scheduler  Successfully assigned nvidia/nvidia-device-plugin-mps-control-daemon-8vqms to k8s-worker-gpu-0
   Normal   Pulled     43s                kubelet            spec.initContainers{toolkit-validation}: Container image "nvcr.io/nvidia/gpu-operator:v26.7.1" already present on machine and can be accessed by the pod
   Normal   Created    43s                kubelet            spec.initContainers{toolkit-validation}: Container created
   Normal   Started    43s                kubelet            spec.initContainers{toolkit-validation}: Container started
   Normal   Pulled     42s                kubelet            spec.initContainers{mps-control-daemon-mounts}: Container image "nvcr.io/nvidia/k8s-device-plugin:v0.20.1" already present on machine and can be accessed by the pod
   Normal   Created    42s                kubelet            spec.initContainers{mps-control-daemon-mounts}: Container created
   Normal   Started    42s                kubelet            spec.initContainers{mps-control-daemon-mounts}: Container started
   Normal   Pulled     41s                kubelet            spec.initContainers{config-manager-init}: Container image "nvcr.io/nvidia/k8s-device-plugin:v0.20.1" already present on machine and can be accessed by the pod
   Normal   Created    41s                kubelet            spec.initContainers{config-manager-init}: Container created
   Normal   Started    41s                kubelet            spec.initContainers{config-manager-init}: Container started
   Normal   Pulled     40s                kubelet            spec.containers{mps-control-daemon-ctr}: Container image "nvcr.io/nvidia/k8s-device-plugin:v0.20.1" already present on machine and can be accessed by the pod
   Normal   Created    40s                kubelet            spec.containers{mps-control-daemon-ctr}: Container created
   Normal   Started    40s                kubelet            spec.containers{mps-control-daemon-ctr}: Container started
   Normal   Pulled     26s (x3 over 40s)  kubelet            spec.containers{config-manager}: Container image "nvcr.io/nvidia/k8s-device-plugin:v0.20.1" already present on machine and can be accessed by the pod
   Normal   Created    26s (x3 over 40s)  kubelet            spec.containers{config-manager}: Container created
   Normal   Started    26s (x3 over 40s)  kubelet            spec.containers{config-manager}: Container started
   Warning  BackOff    26s (x2 over 38s)  kubelet            spec.containers{config-manager}: Back-off restarting failed container config-manager in pod nvidia-device-plugin-mps-control-daemon-8vqms_nvidia(2956c3ac-7f02-4752-a362-2950f79074be)

  • If a pod/ds is in an error state or pending state kubectl logs -n OPERATOR_NAMESPACE POD_NAME --all-containers

     kubectl logs -n nvidia nvidia-device-plugin-mps-control-daemon-8vqms --all-containers
 I0925 16:12:18.620956   72937 main.go:80] "NVIDIA MPS Control Daemon" version=<
 	487d7a93
 	commit: 487d7a93894b0f3ad08097169593972eb3ca8206
  >
 W0925 16:12:19.627450   73059 client_config.go:683] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
 I0925 16:12:19.627665   73059 main.go:248] Waiting for change to 'nvidia.com/device-plugin.config' label
 I0925 16:12:19.627672   73059 main.go:250] Label change detected: nvidia.com/device-plugin.config=
 I0925 16:12:19.627697   73059 main.go:362] No value set. Selecting default name: fullGpu
 I0925 16:12:19.627701   73059 main.go:306] Updating to config: fullGpu
 I0925 16:12:19.627733   73059 main.go:321] Successfully updated to config: fullGpu
 I0925 16:12:20.651387   73188 main.go:80] "NVIDIA MPS Control Daemon" version=<
 	487d7a93
 	commit: 487d7a93894b0f3ad08097169593972eb3ca8206
  >
 I0925 16:12:20.651458   73188 main.go:109] Starting OS watcher.
 I0925 16:12:20.651602   73188 main.go:123] Starting Daemons.
 I0925 16:12:20.651616   73188 main.go:166] Loading configuration.
 I0925 16:12:20.651802   73188 main.go:181] Updating config with default resource matching patterns.
 I0925 16:12:20.651860   73188 main.go:192]
 Running with config:
 {
   "version": "v1",
   "flags": {
     "migStrategy": "none",
     "failOnInitError": null,
     "gdrcopyEnabled": null,
     "gdsEnabled": null,
     "mofedEnabled": null,
     "useNodeFeatureAPI": null,
     "deviceDiscoveryStrategy": null,
     "plugin": {
       "passDeviceSpecs": null,
       "deviceListStrategy": null,
       "deviceIDStrategy": null,
       "cdiAnnotationPrefix": null,
       "nvidiaCTKPath": null,
       "containerDriverRoot": null,
       "sharedDevicesAllocationPolicy": null
     }
   },
   "resources": {
     "gpus": [
       {
         "pattern": "*",
         "name": "nvidia.com/gpu"
       }
     ]
   },
   "sharing": {
     "timeSlicing": {}
   },
   "imex": {}
 }
 I0925 16:12:20.651866   73188 main.go:196] Retrieving MPS daemons.
 I0925 16:12:20.651875   73188 manager.go:66] "Sharing strategy is not MPS; skipping MPS manager creation" strategy="none"
 I0925 16:12:20.651887   73188 main.go:205] No devices are configured for MPS sharing; Waiting indefinitely.
 W0925 16:15:10.260092   74999 client_config.go:683] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
 I0925 16:15:10.260297   74999 main.go:248] Waiting for change to 'nvidia.com/device-plugin.config' label
 I0925 16:15:10.260304   74999 main.go:250] Label change detected: nvidia.com/device-plugin.config=
 I0925 16:15:10.260326   74999 main.go:362] No value set. Selecting default name: fullGpu
 I0925 16:15:10.260331   74999 main.go:306] Updating to config: fullGpu
 I0925 16:15:10.260374   74999 main.go:321] Successfully updated to config: fullGpu
 I0925 16:15:10.260378   74999 main.go:325] Sending signal 'hangup' to '/usr/bin/mps-control-daemon'
 E0925 16:15:10.264400   74999 main.go:210] error finding pid: no process found

  • Output from running nvidia-smi from the driver container: kubectl exec DRIVER_POD_NAME -n OPERATOR_NAMESPACE -c nvidia-driver-ctr -- nvidia-smi

    Fri Sep 25 16:20:15 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.91.07              Driver Version: 595.91.07      CUDA Version: 13.2     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5080        On  |   00000000:00:06.0 Off |                  N/A |
|  0%   39C    P8             16W /  360W |       1MiB /  16303MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 5080        On  |   00000000:00:07.0 Off |                  N/A |
|  0%   38C    P8             11W /  360W |       1MiB /  16303MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA GeForce RTX 5080        On  |   00000000:00:08.0 Off |                  N/A |
|  0%   39C    P8             15W /  360W |       1MiB /  16303MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA GeForce RTX 5080        On  |   00000000:00:09.0 Off |                  N/A |
|  0%   38C    P8             12W /  360W |       1MiB /  16303MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Details
Using AI assistant to debug, I managed to find the following cause:

GPU Operator v26.7.1 defines:

  # assets/state-mps-control-daemon/0400_daemonset.yaml
  command: [mps-control-daemon]

and:

  - name: SEND_SIGNAL
    value: "true"
  - name: SIGNAL
    value: "1"
  - name: PROCESS_TO_SIGNAL
    value: "/usr/bin/mps-control-daemon"

Sources:

config-manager uses an exact command-line match

In k8s-device-plugin v0.20.1, config-manager searches processes and compares the first command-line element exactly:

  if cmdline[0] == f.ProcessToSignal {
      return p.PID, nil
  }

Source:

The same file returns the observed error if no matching process is found:

  return fmt.Errorf("error finding pid: %v", err)

Source:

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugIssue/PR to expose/discuss/fix a bugneeds-triageissue or PR has not been assigned a priority-px label

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions