Skip to content

Commit 9ecef5e

Browse files
un-defclaude
andauthored
Kubernetes: add proxy_jump.node_selector and tolerations (#4459)
The jump pod could land on any node. On some clusters, e.g., Nebius, a NodePort is reachable only on the node that runs the pod, so the jump pod is unreachable if that node has no ExternalIP. Users may also want to keep the jump pod off GPU nodes. * proxy_jump.node_selector restricts the jump pod to labeled nodes. Items are key=value, key= (an empty value), or key (any value). As nodeSelector can't match any value, all items go into a single required node affinity term. * proxy_jump.tolerations items are key[=value][:effect], the kubectl taint format with an optional effect. Without a value, any value is tolerated; without an effect, any effect. An empty key, which would tolerate all taints, is rejected. * If tolerations is not set, GPU node taints are autodetected as before, unless node_selector is set, since autodetection considers all nodes, not only the selected ones. tolerations: [] turns autodetection off. Notes: * New values don't apply to an existing jump pod. The docs describe how to apply them with kubectl. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 4c3b436 commit 9ecef5e

10 files changed

Lines changed: 479 additions & 47 deletions

File tree

‎mkdocs/docs/concepts/backends.md‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1271,12 +1271,32 @@ projects:
12711271

12721272
When the jump pod restarts or is recreated, its SSH sessions drop. The `dstack` server reconnects automatically, but `dstack attach` may fail for runs submitted before that. If reliable SSH access to runs is important, deploy [`dstack-sshproxy`](../guides/server-deployment.md#ssh-proxy).
12731273

1274-
!!! warning "Changing `cpu` and `memory`"
1274+
To run the jump pod only on specific nodes, e.g., nodes with an `ExternalIP` or CPU-only nodes, label them and set `proxy_jump.node_selector`. If these nodes are [tainted](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/), also set `proxy_jump.tolerations`:
1275+
1276+
```yaml
1277+
contexts:
1278+
- name: gpu-cluster-a
1279+
proxy_jump:
1280+
node_selector:
1281+
- example.com/jump-host
1282+
tolerations:
1283+
- example.com/dedicated=jump-host:NoSchedule
1284+
```
1285+
1286+
Each `node_selector` item is `key=value`, `key=` (an empty value), or `key` (any value), and a node must match all of them. Since a `nodeSelector` can't match any value, `dstack` applies `node_selector` as a required [node affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity) term. If none of the selected nodes can run the jump pod, it stays pending and runs can't be provisioned, so label several nodes.
1287+
1288+
Each `tolerations` item is `key[=value][:effect]`, the `kubectl taint` format, so a taint added with `kubectl taint` can be copied as is. Without `=value`, any value is tolerated; without `:effect`, any effect. If `tolerations` isn't set, `dstack` tolerates the GPU node taints when every node is tainted, but not if `node_selector` is set.
1289+
1290+
!!! warning "Changing `cpu`, `memory`, `node_selector`, and `tolerations`"
12751291
New values don't apply to an existing jump pod. To apply them, update the jump pod's ReplicaSet in the context's namespace and delete the pod so that Kubernetes recreates it:
12761292

12771293
```shell
1294+
# cpu and memory
12781295
kubectl -n <namespace> set resources replicaset dstack-<project>-ssh-jump-pod \
12791296
--requests=cpu=1,memory=2Gi --limits=cpu=1,memory=2Gi
1297+
# node_selector and tolerations: edit spec.template.spec.affinity and .tolerations
1298+
kubectl -n <namespace> edit replicaset dstack-<project>-ssh-jump-pod
1299+
# Recreate the pod
12801300
kubectl -n <namespace> delete pod -l app.kubernetes.io/instance=dstack-<project>-ssh-jump-pod
12811301
```
12821302

‎src/dstack/_internal/core/backends/kubernetes/compute.py‎

Lines changed: 76 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,7 @@
6060
OBJECT_NAME_MAX_LENGTH,
6161
AnyKubernetesGPUResource,
6262
KubernetesResource,
63+
Operator,
6364
PodPhase,
6465
ResourceLimits,
6566
ResourceRequests,
@@ -80,7 +81,9 @@
8081
get_registry_auth_secret_name,
8182
is_hard_taint,
8283
is_taint_tolerated,
84+
parse_node_selector_requirement,
8385
parse_quantity,
86+
parse_toleration,
8487
)
8588
from dstack._internal.core.backends.kubernetes.utils import (
8689
LEGACY_CURRENT_CONTEXT_REGION,
@@ -139,11 +142,6 @@
139142
JOB_POD_SCHEDULING_TIMEOUT = 10
140143

141144

142-
class Operator(str, Enum):
143-
EXISTS = "Exists"
144-
IN = "In"
145-
146-
147145
class RestartPolicy(str, Enum):
148146
"""
149147
This policy determines how Kubernetes reacts to containers exiting due to errors
@@ -267,6 +265,8 @@ def run_job(
267265
authorized_keys=project_authorized_keys,
268266
cpu_request_limit=cluster.proxy_jump.cpu or JUMP_POD_DEFAULT_CPU,
269267
memory_request_limit=cluster.proxy_jump.memory or JUMP_POD_DEFAULT_MEMORY,
268+
node_selector=cluster.proxy_jump.node_selector,
269+
tolerations=cluster.proxy_jump.tolerations,
270270
)
271271

272272
pod_name = generate_unique_instance_name_for_job(
@@ -888,6 +888,8 @@ def _create_jump_pod_service_if_not_exists(
888888
authorized_keys: list[str],
889889
cpu_request_limit: str,
890890
memory_request_limit: str,
891+
node_selector: Optional[list[str]],
892+
tolerations: Optional[list[str]],
891893
) -> None:
892894
base_labels = build_base_labels(
893895
component="ssh-proxy",
@@ -959,35 +961,19 @@ def _create_jump_pod_service_if_not_exists(
959961
# Otherwise, the existing legacy pod will be adopted by a newly created ReplicaSet.
960962

961963
if should_create_replica_set:
962-
# False if we found at least one node without any "hard" taint, that is, if we don't need
963-
# to specify the toleration.
964-
toleration_required = True
965-
# (key, effect) pairs.
966-
tolerated_taints: set[tuple[str, str]] = set()
967-
for node in api.list_node().items:
968-
if (node_spec := node.spec) is None:
969-
continue
970-
# True if the node has at least one NoExecute or NoSchedule taint.
971-
has_hard_taint = False
972-
taints = node_spec.taints or []
973-
for taint in taints:
974-
# A "soft" taint, ignore.
975-
if not is_hard_taint(taint):
976-
continue
977-
has_hard_taint = True
978-
if is_taint_tolerated(taint):
979-
tolerated_taints.add((taint.key, taint.effect))
980-
if not has_hard_taint:
981-
toleration_required = False
982-
break
983-
tolerations: list[client.V1Toleration] = []
984-
if toleration_required:
985-
for key, effect in tolerated_taints:
986-
tolerations.append(
987-
client.V1Toleration(key=key, operator=Operator.EXISTS, effect=effect)
988-
)
989-
if not tolerations:
990-
logger.warning("No appropriate node found, the jump pod may never be scheduled")
964+
affinity: Optional[client.V1Affinity] = None
965+
# Not for an empty `node_selector`, as an empty node selector term matches no nodes
966+
if node_selector:
967+
affinity = _get_jump_pod_affinity(node_selector)
968+
pod_tolerations: list[client.V1Toleration]
969+
if tolerations is not None:
970+
pod_tolerations = [parse_toleration(toleration) for toleration in tolerations]
971+
elif node_selector is not None:
972+
# Autodetection considers all nodes, not only the selected ones, so the user
973+
# specifies tolerations for the selected nodes, if any are needed.
974+
pod_tolerations = []
975+
else:
976+
pod_tolerations = _get_jump_pod_tolerations_from_node_taints(api)
991977
commands = _get_jump_pod_commands(authorized_keys)
992978
# Equal CPU and memory requests and limits for the Guaranteed QoS class
993979
limits = {
@@ -1030,7 +1016,8 @@ def _create_jump_pod_service_if_not_exists(
10301016
)
10311017
],
10321018
restart_policy=RestartPolicy.ALWAYS,
1033-
tolerations=tolerations,
1019+
affinity=affinity,
1020+
tolerations=pod_tolerations,
10341021
),
10351022
),
10361023
),
@@ -1220,6 +1207,60 @@ def _check_and_configure_jump_pod_service(
12201207
)
12211208

12221209

1210+
def _get_jump_pod_affinity(node_selector: list[str]) -> client.V1Affinity:
1211+
# A nodeSelector cannot express `Exists`. A single node affinity term matches like
1212+
# a nodeSelector: a node must meet all requirements.
1213+
return client.V1Affinity(
1214+
node_affinity=client.V1NodeAffinity(
1215+
required_during_scheduling_ignored_during_execution=client.V1NodeSelector(
1216+
node_selector_terms=[
1217+
client.V1NodeSelectorTerm(
1218+
match_expressions=[
1219+
parse_node_selector_requirement(requirement)
1220+
for requirement in node_selector
1221+
],
1222+
),
1223+
],
1224+
),
1225+
),
1226+
)
1227+
1228+
1229+
def _get_jump_pod_tolerations_from_node_taints(
1230+
api: client.CoreV1Api,
1231+
) -> list[client.V1Toleration]:
1232+
# False if we found at least one node without any "hard" taint, that is, if we don't need
1233+
# to specify the toleration.
1234+
toleration_required = True
1235+
# (key, effect) pairs.
1236+
tolerated_taints: set[tuple[str, str]] = set()
1237+
for node in api.list_node().items:
1238+
if (node_spec := node.spec) is None:
1239+
continue
1240+
# True if the node has at least one NoExecute or NoSchedule taint.
1241+
has_hard_taint = False
1242+
taints = node_spec.taints or []
1243+
for taint in taints:
1244+
# A "soft" taint, ignore.
1245+
if not is_hard_taint(taint):
1246+
continue
1247+
has_hard_taint = True
1248+
if is_taint_tolerated(taint):
1249+
tolerated_taints.add((taint.key, taint.effect))
1250+
if not has_hard_taint:
1251+
toleration_required = False
1252+
break
1253+
tolerations: list[client.V1Toleration] = []
1254+
if toleration_required:
1255+
for key, effect in tolerated_taints:
1256+
tolerations.append(
1257+
client.V1Toleration(key=key, operator=Operator.EXISTS, effect=effect)
1258+
)
1259+
if not tolerations:
1260+
logger.warning("No appropriate node found, the jump pod may never be scheduled")
1261+
return tolerations
1262+
1263+
12231264
def _get_jump_pod_commands(authorized_keys: list[str]) -> list[str]:
12241265
authorized_keys_content = "\n".join(authorized_keys)
12251266
commands = [

‎src/dstack/_internal/core/backends/kubernetes/configurator.py‎

Lines changed: 21 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,12 @@
1313
KubernetesProxyJumpConfig,
1414
KubernetesStoredConfig,
1515
)
16-
from dstack._internal.core.backends.kubernetes.resources import parse_quantity, validate_quantity
16+
from dstack._internal.core.backends.kubernetes.resources import (
17+
parse_node_selector_requirement,
18+
parse_quantity,
19+
parse_toleration,
20+
validate_quantity,
21+
)
1722
from dstack._internal.core.backends.kubernetes.utils import (
1823
check_cluster,
1924
get_clusters_from_backend_config,
@@ -112,6 +117,7 @@ def _check_config_proxy_jumps(self, config: KubernetesBackendConfig):
112117
def _check_proxy_jump(
113118
self, proxy_jump: KubernetesProxyJumpConfig, *, context_name: Optional[str] = None
114119
):
120+
in_context = f" in context {context_name!r}" if context_name is not None else ""
115121
for field, quantity, min_quantity in [
116122
("cpu", proxy_jump.cpu, _JUMP_POD_MIN_CPU),
117123
("memory", proxy_jump.memory, _JUMP_POD_MIN_MEMORY),
@@ -126,7 +132,17 @@ def _check_proxy_jump(
126132
if parse_quantity(quantity) >= parse_quantity(min_quantity):
127133
continue
128134
reason = f"must be at least {min_quantity}"
129-
msg = f"invalid proxy_jump.{field} value {quantity!r}"
130-
if context_name is not None:
131-
msg = f"{msg} in context {context_name!r}"
132-
raise ServerClientError(f"{msg}: {reason}")
135+
raise ServerClientError(
136+
f"invalid proxy_jump.{field} value {quantity!r}{in_context}: {reason}"
137+
)
138+
for field, items, parse in [
139+
("node_selector", proxy_jump.node_selector, parse_node_selector_requirement),
140+
("tolerations", proxy_jump.tolerations, parse_toleration),
141+
]:
142+
for item in items or []:
143+
try:
144+
parse(item)
145+
except ValueError as e:
146+
raise ServerClientError(
147+
f"invalid proxy_jump.{field} item {item!r}{in_context}: {e}"
148+
)

‎src/dstack/_internal/core/backends/kubernetes/models.py‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,27 @@ class KubernetesProxyJumpConfig(CoreModel):
3838
)
3939
),
4040
] = None
41+
node_selector: Annotated[
42+
Optional[list[str]],
43+
Field(
44+
description=(
45+
"The labels that the jump pod's node must have."
46+
" Each item is `key=value`, `key=` (an empty value), or `key` (any value)"
47+
)
48+
),
49+
] = None
50+
tolerations: Annotated[
51+
Optional[list[str]],
52+
Field(
53+
description=(
54+
"The node taints that the jump pod tolerates."
55+
" Each item is `key[=value][:effect]`, as in `kubectl taint`."
56+
" Without `=value`, any value is tolerated; without `:effect`, any effect."
57+
" If not set, defaults to the GPU node taints when every node is tainted,"
58+
" or to none if `node_selector` is set"
59+
)
60+
),
61+
] = None
4162

4263

4364
class KubernetesContextConfig(CoreModel):

‎src/dstack/_internal/core/backends/kubernetes/resources.py‎

Lines changed: 57 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,14 @@
99

1010
import gpuhunt
1111
from gpuhunt import KNOWN_AMD_GPUS, KNOWN_NVIDIA_GPUS, AcceleratorVendor
12-
from kubernetes.client import CoreV1Api, V1Node, V1ObjectMeta, V1Taint
12+
from kubernetes.client import (
13+
CoreV1Api,
14+
V1Node,
15+
V1NodeSelectorRequirement,
16+
V1ObjectMeta,
17+
V1Taint,
18+
V1Toleration,
19+
)
1320
from kubernetes.utils.quantity import parse_quantity as _parse_quantity
1421
from typing_extensions import Self
1522

@@ -28,7 +35,7 @@
2835
)
2936
from dstack._internal.core.models.resources import Memory, ResourcesSpec
3037
from dstack._internal.utils import docker as docker_utils
31-
from dstack._internal.utils.common import get_or_error
38+
from dstack._internal.utils.common import get_or_error, list_enum_values_for_annotation
3239
from dstack._internal.utils.logging import get_logger
3340

3441
logger = get_logger(__name__)
@@ -93,6 +100,12 @@ class TaintEffect(str, Enum):
93100
PREFER_NO_SCHEDULE = "PreferNoSchedule"
94101

95102

103+
class Operator(str, Enum):
104+
EQUAL = "Equal"
105+
EXISTS = "Exists"
106+
IN = "In"
107+
108+
96109
class KubernetesResource(str, Enum):
97110
CPU = "cpu"
98111
MEMORY = "memory"
@@ -377,6 +390,48 @@ def validate_label_value(value: str) -> None:
377390
raise ValueError("Invalid value")
378391

379392

393+
def parse_node_selector_requirement(requirement: str) -> V1NodeSelectorRequirement:
394+
"""
395+
Parses `key=value`, `key=` (an empty value), or `key` (any value).
396+
"""
397+
key, value_sep, value = requirement.partition("=")
398+
_validate_key(key)
399+
if not value_sep:
400+
return V1NodeSelectorRequirement(key=key, operator=Operator.EXISTS)
401+
validate_label_value(value)
402+
return V1NodeSelectorRequirement(key=key, operator=Operator.IN, values=[value])
403+
404+
405+
def parse_toleration(toleration: str) -> V1Toleration:
406+
"""
407+
Parses `key[=value][:effect]`, the `kubectl taint` format with an optional effect.
408+
Without `=value`, tolerates any value; without `:effect`, tolerates any effect.
409+
"""
410+
key_value, effect_sep, effect = toleration.partition(":")
411+
key, value_sep, value = key_value.partition("=")
412+
# An empty key would tolerate all taints
413+
_validate_key(key)
414+
taint_effect: Optional[TaintEffect] = None
415+
if effect_sep:
416+
try:
417+
taint_effect = TaintEffect(effect)
418+
except ValueError:
419+
raise ValueError(
420+
f"Invalid effect, must be one of: {list_enum_values_for_annotation(TaintEffect)}"
421+
)
422+
if not value_sep:
423+
return V1Toleration(key=key, operator=Operator.EXISTS, effect=taint_effect)
424+
validate_label_value(value)
425+
return V1Toleration(key=key, operator=Operator.EQUAL, value=value, effect=taint_effect)
426+
427+
428+
def _validate_key(key: str) -> None:
429+
try:
430+
validate_label_key(key)
431+
except ValueError as e:
432+
raise ValueError(f"Invalid key: {e}") from e
433+
434+
380435
def build_dockerconfigjson(image_name: str, username: str, password: str) -> str:
381436
registry = docker_utils.parse_image_name(image_name).registry
382437
if registry is None or docker_utils.is_default_registry(registry):

‎src/dstack/_internal/core/compatibility/backends.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,4 +43,8 @@ def _get_kubernetes_proxy_jump_excludes(
4343
excludes["cpu"] = True
4444
if all(proxy_jump.memory is None for proxy_jump in proxy_jumps):
4545
excludes["memory"] = True
46+
if all(proxy_jump.node_selector is None for proxy_jump in proxy_jumps):
47+
excludes["node_selector"] = True
48+
if all(proxy_jump.tolerations is None for proxy_jump in proxy_jumps):
49+
excludes["tolerations"] = True
4650
return excludes

0 commit comments

Comments
 (0)