Repository navigation
Commit e2f9100
authored
Kubernetes: manage the jump pod with a ReplicaSet (#4447)
The jump pod was a bare pod, reused for as long as it existed. Once
evicted, for example due to node pressure, it stayed in the Failed
phase and was never recreated, so every job in the project failed
until someone deleted the pod by hand.
The jump pod is now managed by a ReplicaSet, which replaces the pod
after an eviction or a node removal. Not a Deployment: its ReplicaSet
selects pods by the pod-template-hash label and thus cannot adopt the
existing pod, and rollouts and rollbacks are not needed anyway.
* run_job() creates the ReplicaSet if it doesn't exist. The ReplicaSet
adopts a running legacy pod, a finished one is deleted. A pod created
before #3891 has different labels and is left as is until it's gone.
* A service created before #3891 selects pods by the old labels. Once
the old pod is gone, the service is recreated with the new selector
and the same NodePort, so that ssh_proxy of running jobs stays valid.
* update_provisioning_data() lists jump pods by label and picks a ready
one. It falls back to the name for a pod created before #3891 and
fails right away if that pod is finished, as it's never replaced.
* Creating the namespace, the ReplicaSet, and the service ignores 409
to tolerate concurrent run_job() calls.
Notes:
* The Role now needs get/create on replicasets and list on pods.
Without the update, Kubernetes provisioning fails with 403.
* Once the legacy pod is gone, servers before this commit recreate
a bare pod and the service on each run_job(). The ReplicaSet adopts
and deletes the pod, and the NodePort changes if proxy_jump.port is
not set. Downgrading requires deleting the ReplicaSet.
Part-of: #44071 parent 3b13a7a commit e2f9100
5 files changed
Lines changed: 571 additions & 128 deletions
File tree
- mkdocs/snippets/kubernetes
- src
- dstack/_internal/core/backends/kubernetes
- tests/_internal/core/backends/kubernetes
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | | - | |
| 9 | + | |
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
0 commit comments