Repository navigation
Kubernetes: detect RTX PRO GPUs from nvidia.com/gpu.product - #4492
Conversation
GPU Feature Discovery labels RTX PRO nodes as e.g. NVIDIA-RTX-PRO-6000-Blackwell-Max-Q-Workstation-Edition. The RTX- -> RTX step made that NVIDIA-RTXPRO-6000-..., whose tokens (rtxpro, 6000) never match the known name RTXPRO6000, so such nodes got no GPU offers. Join RTX-PRO- -> RTXPRO first.
peterschmidt85
left a comment
There was a problem hiding this comment.
Thanks for the fix! Could you take a look at the RTX PRO 5000 comment below before we merge?
| mig_profile = mig_resource.removeprefix(NVIDIA_MIG_RESOURCE_PREFIX) | ||
| gpu_product = gpu_product.replace("RTX-", "RTX") | ||
| # Join "RTX" with the model, e.g. "RTX-4090" -> "RTX4090", "RTX-PRO-6000" -> "RTXPRO6000" | ||
| gpu_product = gpu_product.replace("RTX-PRO-", "RTXPRO").replace("RTX-", "RTX") |
There was a problem hiding this comment.
This change also makes the RTX PRO 5000 detectable, but the 48GB card gets the wrong memory:
get_nvidia_gpu_from_node_labels({"nvidia.com/gpu.product": "NVIDIA-RTX-PRO-5000-Blackwell"})
# Gpu(name='RTXPRO5000', memory_mib=73728), i.e. 72GB instead of 48GBSo a run that needs more than 48GB could be scheduled on it. gpuhunt has two RTXPRO5000 entries, 48GB and 72GB, and NVIDIA_GPU_NAME_TO_GPU_INFO keeps the last one. Only the 72GB card has its memory in the product name (NVIDIA RTX PRO 5000 72GB Blackwell).
Could you add this next to the A100 check below, plus test cases for both cards?
# RTX PRO 5000 may come in two variants
if gpu_name == "RTXPRO5000":
gpu_memory = (72 if "72GB" in gpu_product else 48) * 1024There was a problem hiding this comment.
Thanks, confirmed: the driver's supported-GPU list names the cards "NVIDIA RTX PRO 5000 Blackwell" and "NVIDIA RTX PRO 5000 72GB Blackwell" (same device ID), so only the 72GB label carries the size. Added your check next to the A100 one, with test cases for both labels (cd1c911).
gpuhunt lists RTXPRO5000 with 48GB and 72GB, and the name-to-info map keeps the last one, so the 48GB card (NVIDIA-RTX-PRO-5000-Blackwell) was offered with 72GB. Only the 72GB card has its memory in the product name (NVIDIA RTX PRO 5000 72GB Blackwell).
Problem
On a Kubernetes node with RTX PRO 6000 Blackwell GPUs, the
kubernetesbackend offers no GPUs. NVIDIA GPU Feature Discovery (k8s-device-plugin v0.20.1) labels the node:and
nvidia.com/gpu: 2is allocatable, but:dstack offerwithout--gpulists the node ascpu=x86:1only.In
get_nvidia_gpu_from_node_labels,replace("RTX-", "RTX")turns the label intoNVIDIA-RTXPRO-6000-Blackwell-..., whose tokens arertxpro,6000, ... None equals the known nameRTXPRO6000, so the function returnsNone:Reproduced with 0.22.8 against a single-node RKE2 v1.36.6 cluster; same code on
master.Fix
Replace
RTX-PRO-withRTXPRObefore the existingRTX-→RTXstep, so the label has the tokenrtxpro6000. Other products are unaffected: only labels containingRTX-PRO-change.The RTX PRO 5000 comes in 48GB and 72GB variants with the same name, and only the 72GB one has its memory in the product name (
NVIDIA RTX PRO 5000 72GB Blackwell). Its memory is 72GB if the label contains72GB, and 48GB otherwise.With the change applied to 0.22.6 on the same cluster,
dstack offer --gpu 1..listsgpu=RTXPRO6000:96GB:1.Tests
test_returns_rtx_gpuinsrc/tests/_internal/core/backends/kubernetes/test_resources.py: RTX PRO 6000 Max-Q (the label from the node above), RTX PRO 6000 Server Edition, RTX PRO 5000 (48GB and 72GB), RTX PRO 4500, and GeForce RTX 4090 (unchanged behaviour). The RTX PRO cases fail without the fix.uv run ruff check --fix,uv run ruff format: no changesuv run pytest src/tests: 5129 passed, 1543 skipped