Skip to content

Enforce singleton GPUCluster name and select ClusterPolicy by creation time - #2689

Merged
karthikvetrivel merged 1 commit into
mainfrom
kv-singleton-resolution
Jul 28, 2026
Merged

Enforce singleton GPUCluster name and select ClusterPolicy by creation time#2689
karthikvetrivel merged 1 commit into
mainfrom
kv-singleton-resolution

Conversation

@karthikvetrivel

@karthikvetrivel karthikvetrivel commented Jul 28, 2026

Copy link
Copy Markdown
Member

With multiple CRs of a kind, controllers resolved "the active one" inconsistently: the owning controllers kept a first-reconciled-wins claim in memory (unstable across operator restarts, invisible to other controllers), while resolveActiveConfig picked the first list item. This PR makes the singleton deterministic for both kinds:

  • GPUCluster: a CEL rule on the CRD pins metadata.name to gpu-cluster (the name the Helm chart creates), enforcing the singleton at admission. That makes the controller's runtime singleton handling dead code, so it is removed: the in-memory first-wins claim and the ignored status value are gone, and the sample is renamed to gpu-cluster. GPUCluster has not shipped, so there is no upgrade path to consider.
  • ClusterPolicy: nothing enforces a singleton at admission, so the instance with the oldest creationTimestamp (name as tie-breaker) is treated as the singleton, via a new getSingletonClusterPolicy helper used by the ClusterPolicy controller, resolveActiveConfig, and the GPUCluster prerequisite check (which now inspects only the singleton ClusterPolicy). This selection is a heuristic: ideally the operator would hold a reference to the singleton ClusterPolicy globally throughout its lifetime, and until it does, picking the oldest is the next-best approach.

Deleting the singleton ClusterPolicy promotes the next-oldest lazily (on its next reconcile), same as today.

@karthikvetrivel
karthikvetrivel force-pushed the kv-singleton-resolution branch from 2e74208 to 3177a01 Compare July 28, 2026 19:23
@karthikvetrivel
karthikvetrivel marked this pull request as ready for review July 28, 2026 19:28
Base automatically changed from kv-nvidiadriver-standalone to main July 28, 2026 20:23
Comment thread controllers/active_config.go Outdated
@karthikvetrivel
karthikvetrivel force-pushed the kv-singleton-resolution branch from 3177a01 to c17b3f0 Compare July 28, 2026 20:44
Comment thread controllers/active_config.go Outdated
Comment thread api/nvidia/v1alpha1/gpucluster_types.go
@karthikvetrivel
karthikvetrivel force-pushed the kv-singleton-resolution branch 2 times, most recently from 0f2c0a6 to dc6f211 Compare July 28, 2026 21:14
@tariq1890

Copy link
Copy Markdown
Contributor

@karthikvetrivel Can we amend the PR title and commit as well?

Note, the reason I am being particular here is that we don't really elect a ClusterPolicy in the way that distributed systems elect a "leader" or "primary actor". Secondly, let'd mention in the comments that the selection of the oldest clusterPolicy as the singleton is a heuristic. Ideally, we should have a way of holding a reference to the singleton ClusterPolicy globally throughout the operator's lifetime. Since we don't have this right now, selection of the ClusterPolicy by the oldestTimestamp is the next best approach.

Comment thread controllers/active_config.go Outdated
Comment thread controllers/active_config.go Outdated
Comment thread controllers/active_config.go Outdated
…n time

Signed-off-by: Karthik Vetrivel <kvetrivel@nvidia.com>
@karthikvetrivel
karthikvetrivel force-pushed the kv-singleton-resolution branch from dc6f211 to a6f6dbf Compare July 28, 2026 21:37
@karthikvetrivel karthikvetrivel changed the title Enforce singleton GPUCluster name and elect ClusterPolicy by creation time Enforce singleton GPUCluster name and select ClusterPolicy by creation time Jul 28, 2026
@karthikvetrivel
karthikvetrivel merged commit 0fd1f0d into main Jul 28, 2026
19 of 21 checks passed
@tariq1890
tariq1890 deleted the kv-singleton-resolution branch July 28, 2026 23:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants