feat: slot-release hibernation for hibernatePolicy=release CocoonSets#23
Merged
Conversation
tonicmuroq
force-pushed
the
feat/hibernate-slot-policy
branch
2 times, most recently
from
July 23, 2026 13:03
ed5bbe7 to
e47b4cc
Compare
CMGS
force-pushed
the
feat/hibernate-slot-policy
branch
7 times, most recently
from
July 24, 2026 16:08
b700d38 to
1c7e9bc
Compare
A hibernated VM's placeholder pod keeps its requests bound to the node,
so the scheduler counts the seat as occupied even though vk-cocoon has
pushed the VM to the registry and destroyed it locally. For
hibernatePolicy=release sets the operator now:
- suspend: after every managed VM's :hibernate snapshot is verified in
the registry (the existing allOwnedPodsHibernated gate), stash the vm
names (delete-time tag GC) and the main pod's node (wake hint), then
delete the owned pods — the seat frees and Suspended lands in the
same pass as the durable receipt that the deletes were issued.
Terminal pods are kept for triage. Fails closed without a registry.
- wake: reconcileWake owns the reconcile of a set coming back from a
pod-less Suspended state. It recreates the main pod unpinned with
restore-from-hibernate and a preferred (soft) affinity to the
hibernated-on node, persists the Waking phase before the create,
surfaces Unschedulable as WakeNoCapacity events, and drops the
:hibernate tag only once the restored VM is live. Gated on phase,
not the current policy. Durable node-hint evidence closes the
informer races a fast unsuspend can hit: a pod-less Suspending set
(crash before the receipt) still restores, a visible main under
Suspending/Suspended is confirmed against the uncached API — stale
view of the delete waits it out, a never-deleted pod wakes in place —
and a restore-marked, non-hibernating main re-engages completion
across a stale phase read; completion clears the hint.
Migration is unaffected and runs first; a pinned released set resumes
through its aborted-migration branch, so the pin always wins over
wake-anywhere. New metrics: slot_release_pods_deleted_total,
slot_release_wake_total{placement=hint-node|pool},
slot_release_wake_unschedulable_total.
cocoon-common is pinned to the spec.hibernatePolicy merge commit.
CMGS
force-pushed
the
feat/hibernate-slot-policy
branch
from
July 24, 2026 16:19
1c7e9bc to
6ffa5fa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A hibernated VM's placeholder pod keeps its CPU/memory requests bound to the node, so the scheduler counts the seat as occupied even though vk-cocoon has pushed the VM to the registry and destroyed it locally. On a node with 60/80 VMs hibernated, 60 seats are dead weight: no new VM can take them and the live VMs run effectively un-oversold.
hibernatePolicy=releaseturns hibernation into "give the seat back to the pool" — wake takes any free seat on any node, exactly like a GCE create can hit a zone stockout.Design: https://github.com/cocoonstack/cocoon-specs/pull/5. Depends on cocoon-common: cocoonstack/cocoon-common#7 (merged; go.mod pinned to its merge commit 4363e4f).
How
All new logic in
cocoonset/slotrelease.go, two insertions inReconcile:reconcileSuspendRelease(replacesreconcileSuspendfor release sets): applies the hibernate annotation as before, then — only after the existingallOwnedPodsHibernatedgate (lifecycle-state=hibernated ∧ observed-generation ≥ CR generation ∧ snapshot present in registry) — stashes the vm names (reusing the delete-time GC stash) and the main pod's node (wake hint) onto the CR, and deletes the owned pods. Terminal pods are kept for the existing post-unsuspend triage. Fails closed without a registry: the seat is never released unverified.reconcileWake: owns the reconcile of a pod-less Suspended set coming back. Persists the newWakingphase before creating the restore pod (crash re-entry resumes the restore instead of falling through tocreateMainAgent, whose restore intent only covers CocoonHibernation CRs and would fresh-boot over the snapshot). The pod is created unpinned withrestore-from-hibernateand a preferred (soft) affinity to the hibernated-on node — old seat still free ⇒ warm wake from the node-local snapshot, otherwise the scheduler places it anywhere in the pool and vk pulls from the registry. Unschedulable surfaces asWakeNoCapacityevents + a metric while the pod keeps waiting. Suspended lands in the same pass as the deletes (the durable receipt); a fast unsuspend that still sees the old main in a stale cache is confirmed against the uncached API (stale view of the delete → wait, never-deleted pod → in-place wake), and a crash before the receipt leaves pod-less Suspending which the node-hint arm restores — no informer ordering can reach a fresh boot over the snapshot. The:hibernatetag is dropped only once the restored VM is live (same rule migration enforces). Gated on phase, never on the current policy or bare tag presence, so a policy edit made while suspended still restores correctly and a stale tag can never roll a running VM back.Migration is unaffected and runs first: a pinned released set resumes through migration's aborted-migration branch, so a
spec.nodeNamepin always wins over wake-anywhere.Metrics
slot_release_pods_deleted_total,slot_release_wake_total{placement=hint-node|pool}(soft-affinity hit rate — note every release wake restores via registry pull: vk drops local snapshots with the pod, so hint-node placement is about future warm-path potential, not a warm restore today),slot_release_wake_unschedulable_total(the out-of-stock signal for fleet sizing).Tests
21 new cases in
cocoonset/slotrelease_test.go(fake-client style, mirroringmigrate_test.go): verify-before-delete ordering, stash-before-delete, terminal-pod retention, restore+preferred-affinity pod shape, pin-wins-over-hint, fresh-boot fallthrough when no snapshot exists, fail-closed probe errors, tag survival until live, Running-phase tag distrust, and normal-flow handoff for retain placeholders. Fullgo test ./...green.