Skip to content

Kubernetes: detect RTX PRO GPUs from nvidia.com/gpu.product - #4492

Merged
peterschmidt85 merged 2 commits into
dstackai:masterfrom
andweng:fix-k8s-rtx-pro-gpu-labels
Nov 16, 2026
Merged

peterschmidt85 merged 2 commits into
dstackai:masterfrom
andweng:fix-k8s-rtx-pro-gpu-labels

Conversation

@andweng

@andweng andweng commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem

On a Kubernetes node with RTX PRO 6000 Blackwell GPUs, the kubernetes backend offers no GPUs. NVIDIA GPU Feature Discovery (k8s-device-plugin v0.20.1) labels the node:

nvidia.com/gpu.product=NVIDIA-RTX-PRO-6000-Blackwell-Max-Q-Workstation-Edition
nvidia.com/gpu.count=2

and nvidia.com/gpu: 2 is allocatable, but:

$ dstack offer --gpu 1..
...
No matching instance offers available. Possible reasons:
https://dstack.ai/docs/guides/troubleshooting/#no-offers

dstack offer without --gpu lists the node as cpu=x86:1 only.

In get_nvidia_gpu_from_node_labels, replace("RTX-", "RTX") turns the label into NVIDIA-RTXPRO-6000-Blackwell-..., whose tokens are rtxpro, 6000, ... None equals the known name RTXPRO6000, so the function returns None:

get_nvidia_gpu_from_node_labels(
    {"nvidia.com/gpu.product": "NVIDIA-RTX-PRO-6000-Blackwell-Max-Q-Workstation-Edition"}
)
# None

Reproduced with 0.22.8 against a single-node RKE2 v1.36.6 cluster; same code on master.

Fix

Replace RTX-PRO- with RTXPRO before the existing RTX- → RTX step, so the label has the token rtxpro6000. Other products are unaffected: only labels containing RTX-PRO- change.

The RTX PRO 5000 comes in 48GB and 72GB variants with the same name, and only the 72GB one has its memory in the product name (NVIDIA RTX PRO 5000 72GB Blackwell). Its memory is 72GB if the label contains 72GB, and 48GB otherwise.

With the change applied to 0.22.6 on the same cluster, dstack offer --gpu 1.. lists gpu=RTXPRO6000:96GB:1.

Tests

test_returns_rtx_gpu in src/tests/_internal/core/backends/kubernetes/test_resources.py: RTX PRO 6000 Max-Q (the label from the node above), RTX PRO 6000 Server Edition, RTX PRO 5000 (48GB and 72GB), RTX PRO 4500, and GeForce RTX 4090 (unchanged behaviour). The RTX PRO cases fail without the fix.

  • uv run ruff check --fix, uv run ruff format: no changes
  • uv run pytest src/tests: 5129 passed, 1543 skipped

GPU Feature Discovery labels RTX PRO nodes as e.g. NVIDIA-RTX-PRO-6000-Blackwell-Max-Q-Workstation-Edition. The RTX- -> RTX step made that NVIDIA-RTXPRO-6000-..., whose tokens (rtxpro, 6000) never match the known name RTXPRO6000, so such nodes got no GPU offers. Join RTX-PRO- -> RTXPRO first.
@andweng
andweng marked this pull request as ready for review November 15, 2026 21:17

@peterschmidt85 peterschmidt85 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix! Could you take a look at the RTX PRO 5000 comment below before we merge?

mig_profile = mig_resource.removeprefix(NVIDIA_MIG_RESOURCE_PREFIX)
gpu_product = gpu_product.replace("RTX-", "RTX")
# Join "RTX" with the model, e.g. "RTX-4090" -> "RTX4090", "RTX-PRO-6000" -> "RTXPRO6000"
gpu_product = gpu_product.replace("RTX-PRO-", "RTXPRO").replace("RTX-", "RTX")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change also makes the RTX PRO 5000 detectable, but the 48GB card gets the wrong memory:

get_nvidia_gpu_from_node_labels({"nvidia.com/gpu.product": "NVIDIA-RTX-PRO-5000-Blackwell"})
# Gpu(name='RTXPRO5000', memory_mib=73728), i.e. 72GB instead of 48GB

So a run that needs more than 48GB could be scheduled on it. gpuhunt has two RTXPRO5000 entries, 48GB and 72GB, and NVIDIA_GPU_NAME_TO_GPU_INFO keeps the last one. Only the 72GB card has its memory in the product name (NVIDIA RTX PRO 5000 72GB Blackwell).

Could you add this next to the A100 check below, plus test cases for both cards?

    # RTX PRO 5000 may come in two variants
    if gpu_name == "RTXPRO5000":
        gpu_memory = (72 if "72GB" in gpu_product else 48) * 1024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, confirmed: the driver's supported-GPU list names the cards "NVIDIA RTX PRO 5000 Blackwell" and "NVIDIA RTX PRO 5000 72GB Blackwell" (same device ID), so only the 72GB label carries the size. Added your check next to the A100 one, with test cases for both labels (cd1c911).

gpuhunt lists RTXPRO5000 with 48GB and 72GB, and the name-to-info map keeps
the last one, so the 48GB card (NVIDIA-RTX-PRO-5000-Blackwell) was offered
with 72GB. Only the 72GB card has its memory in the product name
(NVIDIA RTX PRO 5000 72GB Blackwell).
@peterschmidt85
peterschmidt85 merged commit ff60b85 into dstackai:master Nov 16, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants