diff --git a/docs/design/2026_04_29_partial_data_at_rest_encryption.md b/docs/design/2026_04_29_partial_data_at_rest_encryption.md index d454874a4..65a039fed 100644 --- a/docs/design/2026_04_29_partial_data_at_rest_encryption.md +++ b/docs/design/2026_04_29_partial_data_at_rest_encryption.md @@ -1,6 +1,6 @@ # Data-at-rest encryption for elastickv -Status: Partial — Stages 0–5 and Stage 8 shipped (5E deferred); Stage 6 partially shipped; Stages 7 and 9 open +Status: Partial — Stages 0–5 and Stage 8 shipped (5E deferred); Stage 6 partially shipped; Stage 7 partially shipped; Stage 9 open Author: bootjp Date: 2026-04-29 @@ -30,7 +30,7 @@ Date: 2026-04-29 | 6D | §6.6 `enable-storage-envelope` admin RPC + §7.1 Phase-1 storage cutover (§6.2 toggle ON) + Voters ∪ Learners capability gate (depends on 6B for mutator wiring AND 6C-1+6C-2 for §9.1 startup-refusal guards; bundles 6C-3 for the membership-view-dependent collision check — see rationale) | open | — | | 6E | §6.6 `enable-raft-envelope` admin RPC + §7.1 Phase-2 raft cutover + `raft_envelope_cutover_index` sidecar record + `internal/raftengine/etcd/engine.go` `applyNormalEntry` unwrap hook activation + `ErrRaftUnwrapFailed` HaltApply path + `kv/coordinator.go` / `kv/sharded_coordinator.go` wrap-on-propose switch (Phase-2 leader-side §6.3 proposal-payload wrap) + §7.1 steps 1–6 proposal quiescence barrier (block new user proposal intake, drain in-flight queue, source-tag exemption for the cutover entry itself) + production runtime wiring for startup/apply-time wrap install and post-cutover admin proposals + 6C-4 fail-closed guards. | shipped | 2026_05_31_implemented_6e_enable_raft_envelope.md | | 6F | §6.5 `--encryption-rotate-on-startup` request flag + leader-elected rotation proposal on the default encryption Raft group. The leader rotates every active DEK purpose once before listeners open; followers keep an in-memory pending request and fire only if they acquire leadership in the same process uptime. | shipped | this PR | -| 7 | Writer registry + deterministic nonce (§4.1) | open | — | +| 7 | Writer registry + deterministic nonce (§4.1). Implemented slices: process-start registration, storage-layer registration gate, runtime re-registration for cutover and rotation, and conf-change-time pre-registration. Still open: §5.5 registry projection in `GetSidecarState` / `ResyncSidecar` (`WriterRegistryForCaller`). | partial | `2026_05_26_implemented_7a_process_start_registration.md` + `2026_05_26_implemented_7a2_storage_layer_registration_enforcement.md` + `2026_05_28_implemented_7b_runtime_reregistration.md` + `2026_05_28_implemented_7b_prime_runtime_reregistration_rotation.md` + `2026_05_29_implemented_7c_confchange_time_registration.md` | | 8 | Snapshot header v2 (§4.4); WAL coverage closure (§4.3 / §4.6) | shipped | [`2026_05_29_implemented_8a_snapshot_header_v2.md`](2026_05_29_implemented_8a_snapshot_header_v2.md) + [`2026_06_01_implemented_8b_wal_coverage_closure.md`](2026_06_01_implemented_8b_wal_coverage_closure.md) | | 9 | KMS-backed wrappers, compression, rotation/retire/rewrite, Jepsen (§5.2, §5.4, §6.4, §8) | open | — | diff --git a/docs/design/2026_05_26_proposed_7a2_storage_layer_registration_enforcement.md b/docs/design/2026_05_26_implemented_7a2_storage_layer_registration_enforcement.md similarity index 96% rename from docs/design/2026_05_26_proposed_7a2_storage_layer_registration_enforcement.md rename to docs/design/2026_05_26_implemented_7a2_storage_layer_registration_enforcement.md index 2f8415b95..539dcea04 100644 --- a/docs/design/2026_05_26_proposed_7a2_storage_layer_registration_enforcement.md +++ b/docs/design/2026_05_26_implemented_7a2_storage_layer_registration_enforcement.md @@ -2,11 +2,20 @@ | Field | Value | |---|---| -| Status | proposed | +| Status | implemented | | Date | 2026-05-26 | -| Parent designs | [`2026_04_29_partial_data_at_rest_encryption.md`](2026_04_29_partial_data_at_rest_encryption.md) (§4.1 writer registry), [`2026_05_26_proposed_7a_process_start_registration.md`](2026_05_26_proposed_7a_process_start_registration.md) (7a coordinator-layer gate) | +| Implemented | 2026-07-07 status audit; code landed before this doc promotion | +| Parent designs | [`2026_04_29_partial_data_at_rest_encryption.md`](2026_04_29_partial_data_at_rest_encryption.md) (§4.1 writer registry), [`2026_05_26_implemented_7a_process_start_registration.md`](2026_05_26_implemented_7a_process_start_registration.md) (7a coordinator-layer gate) | | Builds on | 7a (coordinator-layer first-write barrier), 6D-6c-1 (`encryption.StateCache`) | +## Implementation audit + +Implemented by `internal/encryption.StateCache.Registered`, +`StateCache.MarkRegistered`, `store.WithStorageRegistrationGate`, +`store.ErrWriterNotRegistered`, and the direct-vs-FSM write-path flag in +`store`. +The gate is wired from `main_encryption_write_wiring.go`. + ## 0. Why this slice exists 7a's coordinator-layer barrier (`ShardedCoordinator.Dispatch` + @@ -270,7 +279,8 @@ forwarder callers before merge. the first direct-path write does not fail-closed (codex P1).** ### Out of scope -- 7b (post-rotation re-registration), 7c (ConfChange-time registration). +- 7b (post-rotation re-registration), 7c (ConfChange-time registration); + both are implemented by later slices. - Re-evaluating whether route-catalog data *should* be encrypted at all (it is, incidentally, when the envelope is active; changing that is a separate question). diff --git a/docs/design/2026_05_26_proposed_7a_process_start_registration.md b/docs/design/2026_05_26_implemented_7a_process_start_registration.md similarity index 97% rename from docs/design/2026_05_26_proposed_7a_process_start_registration.md rename to docs/design/2026_05_26_implemented_7a_process_start_registration.md index 089e46569..6822b125d 100644 --- a/docs/design/2026_05_26_proposed_7a_process_start_registration.md +++ b/docs/design/2026_05_26_implemented_7a_process_start_registration.md @@ -2,11 +2,20 @@ | Field | Value | |---|---| -| Status | proposed | +| Status | implemented | | Date | 2026-05-26 | +| Implemented | 2026-07-07 status audit; code landed before this doc promotion | | Parent design | [`2026_04_29_partial_data_at_rest_encryption.md`](2026_04_29_partial_data_at_rest_encryption.md) (§4.1 writer registry, §5.2 process-start path) | | Builds on | 6A (`ApplyRegistration` §4.1 case 1–4 dispatch), 5B (`RegisterEncryptionWriter` RPC + leader Proposer wiring), 6C-3 (`ErrLocalEpochRollback`), 6D-6c-2 (`BumpLocalEpoch`, `DeterministicNonceFactory`, keystore hydration) | -| Slice of | Stage 7 (writer registry + deterministic nonce). 7b (post-rotation re-registration) and 7c (ConfChange-time registration) follow. | +| Slice of | Stage 7 (writer registry + deterministic nonce). 7b (post-rotation re-registration) and 7c (ConfChange-time registration) are implemented. | + +## Implementation audit + +Implemented in `main_encryption_registration.go`, `kv/sharded_coordinator.go`, +and the writer-registry helpers in `store` / `internal/encryption`. +Tests live in `kv/sharded_coordinator_registration_gate_test.go`, +`main_encryption_registration_test.go`, and encryption/store registration +gate suites. ## 0. Why this slice exists diff --git a/docs/design/2026_05_28_proposed_7b_prime_runtime_reregistration_rotation.md b/docs/design/2026_05_28_implemented_7b_prime_runtime_reregistration_rotation.md similarity index 86% rename from docs/design/2026_05_28_proposed_7b_prime_runtime_reregistration_rotation.md rename to docs/design/2026_05_28_implemented_7b_prime_runtime_reregistration_rotation.md index 14f060495..13397968e 100644 --- a/docs/design/2026_05_28_proposed_7b_prime_runtime_reregistration_rotation.md +++ b/docs/design/2026_05_28_implemented_7b_prime_runtime_reregistration_rotation.md @@ -2,10 +2,20 @@ | Field | Value | |---|---| -| Status | proposed | +| Status | implemented | | Date | 2026-05-28 | | Parent designs | [`2026_04_29_partial_data_at_rest_encryption.md`](2026_04_29_partial_data_at_rest_encryption.md) (§4.1 writer registry, §5.2 RotateDEK apply), [`2026_05_28_implemented_7b_runtime_reregistration.md`](2026_05_28_implemented_7b_runtime_reregistration.md) (cutover case + §6 deferred-rotation analysis) | | Builds on | 7a (process-start propose path), 7a-2 (storage-layer `Registered()` gate), 7b (runtime watcher for the cutover case) | +| Implemented | 2026-07-07 status audit; code landed before this doc promotion | + +## Implementation audit + +Implemented by `internal/encryption.WithLocalEpoch`, +`internal/encryption.WithRaftLocalEpoch`, `writeRotationSidecar`, +`main_encryption_registration.go`'s runtime registration watcher, and +the per-DEK `StateCache.Registered` gate. Tests cover storage and raft +rotation-local-epoch sidecar writes plus runtime rotation registration +scope/verify paths. ## 0. Why this slice exists @@ -110,40 +120,38 @@ cleaner but ships materially more code; we will revisit it if option ### 3.1 `applyRotateDEK` per-node sidecar mutation -`applyRotateDEK` currently writes the new key with `LocalEpoch: 0` -via `writeRotationSidecar`. The change: - -- Inject a `localEpoch uint16` value into the `Applier` via a new - `WithLocalEpoch(uint16)` functional option (symmetric with - `WithStateCache`, `WithSidecarPath`, etc.). -- The value is `encryptionWriteWiring.epoch` — the same pinned - `w.epoch` the nonce factory is constructed with at process start. - `main_encryption_write_wiring.go`'s `buildEncryptionWriteWiring` - already computes this value; this slice exposes it to the - `Applier` construction site in `main.go`. -- `writeRotationSidecar` reads `a.localEpoch` (defaulting to `0` - when the option was not supplied — preserving the FSM-internal - test harnesses that construct an `Applier` without write-path - state) and writes that value into `sc.Keys[newDEK].LocalEpoch` - **only when `p.Purpose == PurposeStorage`**. - -#### 3.1.1 Why the value is storage-only — Raft DEK rotations must keep `LocalEpoch: 0` +Before this slice, `applyRotateDEK` wrote the new key with +`LocalEpoch: 0` via `writeRotationSidecar`. The change: + +- Inject purpose-specific epoch values into the `Applier` via + `WithLocalEpoch(uint16)` for storage and + `WithRaftLocalEpoch(uint16)` for raft-envelope writes (symmetric + with `WithStateCache`, `WithSidecarPath`, etc.). +- The values are `encryptionWriteWiring.epoch` and + `encryptionWriteWiring.raftEpoch` — the pinned epochs the storage + and raft-envelope nonce factories are constructed with at process + start. `main_encryption_write_wiring.go`'s + `buildEncryptionWriteWiring` computes both values; this slice + exposes them to the `Applier` construction site in `main.go`. +- `writeRotationSidecar` reads `a.localEpoch` for + `PurposeStorage` and `a.raftLocalEpoch` for `PurposeRaft` + (defaulting to `0` when the option was not supplied — preserving + FSM-internal test harnesses that construct an `Applier` without + write-path state) and writes that value into + `sc.Keys[newDEK].LocalEpoch`. + +#### 3.1.1 Why the value is purpose-specific `applyRotateDEK` handles both `PurposeStorage` and `PurposeRaft` rotations today (`writeRotationSidecar` dispatches on `p.Purpose` when assigning `sc.Active.{Storage,Raft}`). The `encryptionWriteWiring.epoch` value piped in by `WithLocalEpoch` is -the **storage** write-path's pinned `w.epoch` — the value the -deterministic nonce factory under the storage envelope is using. - -A raft DEK rotation has its own (future) per-purpose epoch -counter; cross-applying the storage nonce factory's epoch to a -raft DEK's `LocalEpoch` would corrupt the raft DEK's counter -before the raft envelope path consumes it (codex P2 on PR #855). -Until raft envelope support lands with its own per-purpose -plumbing, raft DEK rotations MUST continue to write `LocalEpoch: -0` exactly as today — `writeRotationSidecar`'s switch on -`p.Purpose` gates the new behaviour to storage: +the **storage** write-path's pinned `w.epoch`; the +`encryptionWriteWiring.raftEpoch` value piped in by +`WithRaftLocalEpoch` is the raft-envelope counterpart. Cross-applying +the storage nonce factory's epoch to a raft DEK's `LocalEpoch` would +corrupt the raft counter, so `writeRotationSidecar` selects the +matching epoch for each purpose: ```go switch p.Purpose { @@ -152,7 +160,7 @@ case fsmwire.PurposeStorage: keyLocalEpoch = a.localEpoch case fsmwire.PurposeRaft: sc.Active.Raft = p.DEKID - keyLocalEpoch = 0 // current behaviour preserved + keyLocalEpoch = a.raftLocalEpoch } ``` @@ -290,14 +298,15 @@ separate slice can introduce a per-DEK cache. 1. KEK-unwrap new DEK. 2. Keystore set (`a.keystore.Set(p.DEKID, dek)`). 3. **Sidecar write — with `a.localEpoch` for `Keys[newDEK].LocalEpoch` - when `p.Purpose == PurposeStorage`, `0` otherwise** (see §3.1.1). + when `p.Purpose == PurposeStorage`, and `a.raftLocalEpoch` when + `p.Purpose == PurposeRaft`** (see §3.1.1). 4. **Proposer registration** (`ApplyRegistration(p.ProposerRegistration)`) — proposer's value is `(newDEK, proposer.node_id, proposer.w.epoch)`, set by the proposer at propose time. Verbatim into the writer registry. -Step 3 changes from `LocalEpoch: 0` to `LocalEpoch: a.localEpoch` (for -storage). Step 4 is unchanged. +Step 3 changes from `LocalEpoch: 0` to the purpose-specific pinned +epoch. Step 4 is unchanged. **Sidecar-before-registration is the existing `applyRotateDEK` ordering** (coderabbitai major review on PR #855 raised concern @@ -362,15 +371,21 @@ node's sidecar now records its own emissions accurately. ## 5. Verification action items (for the implementation PR) 1. New `applier_test.go` cases: - - `TestApplier_ApplyRotateDEK_LocalEpochProviderReadAtApply`: - given a `WithLocalEpoch` provider returning 7, the rotation + - `TestApplyRotateDEK_LocalEpochAppliedToStorageKey`: + given `WithLocalEpoch(7)`, the storage rotation entry under a fresh DEK writes `Keys[newDEK].LocalEpoch = 7` to the sidecar. - - `TestApplier_ApplyRotateDEK_NoLocalEpochProviderFallsBack`: + - `TestApplyRotateDEK_RaftLocalEpochAppliedToRaftKey`: + given `WithRaftLocalEpoch(9)`, the raft rotation entry writes + `Keys[newDEK].LocalEpoch = 9` to the raft sidecar key. + - `TestApplyRotateDEK_LocalEpoch_RaftRotationKeepsZero`: + constructing the `Applier` without `WithRaftLocalEpoch` keeps + raft rotations at `LocalEpoch: 0` for unwired harnesses. + - `TestApplyRotateDEK_LocalEpoch_NoOptionFallsBackToZero`: constructing the `Applier` without the option preserves today's `LocalEpoch: 0` behaviour (test-harness backward-compatibility). - - `TestApplier_ApplyRotateDEK_LocalEpochAppliedPerApply`: + - `TestApplyRotateDEK_LocalEpoch_AppliedPerApply`: verify the static `a.localEpoch` value is correctly read on each FSM apply call, not inadvertently reset or derived from mutable state. A multi-apply test fixture (two @@ -465,18 +480,14 @@ mitigations close this gap, in order of operator friction: metrics can warn if the cluster reports mixed versions and a `RotateDEK` is observed in the audit log. -2. **Admin RPC capability gate (middle friction).** The - `EncryptionAdmin.RotateDEK` RPC handler (in the propose-side - mutator) gates on a cluster-wide capability probe — gossiped or - queried via `Distribution.ListNodes` — refusing to propose if - any peer advertises a version older than 7b'. This is - server-side and cannot be bypassed by a misconfigured client. - Failure surface: the RPC returns - `FailedPrecondition: cluster contains pre-7b' nodes; complete - the rolling upgrade before rotating`. The probe is best-effort - (a freshly-joined node could race the check), but the §4.1 - case-2 idempotent posture means a false-positive admit only - degrades availability of the pre-7b' node — it never corrupts. +2. **Admin RPC capability gate (middle friction; not shipped in this + slice).** A future `EncryptionAdmin.RotateDEK` RPC handler can gate + on a cluster-wide capability probe — gossiped or queried via + `Distribution.ListNodes` — refusing to propose if any peer + advertises a version older than 7b'. The current handler performs + leader/proposer/input validation but does not fan out this + capability probe, so this slice relies on the operational guideline + in (1) rather than a server-side mixed-version block. 3. **Cluster-version Raft entry (highest friction; deferred).** A future slice could land a `ClusterVersion` Raft entry that @@ -485,13 +496,15 @@ mitigations close this gap, in order of operator friction: strongest form but adds a new wire entry and a coordination protocol — overkill for a single-feature gate. -7b' will ship (1) and (2). (3) is explicitly deferred — it has +7b' ships (1). (2) and (3) are explicitly deferred — both have broader applicability than just this slice (snapshot v2 in Stage 8 -will face the same problem) and deserves its own design. +will face the same problem) and deserve their own implementation +designs. Roll-forward path: deploy 7b' to every node in the cluster, verify -the capability probe reports all-7b', then issue `RotateDEK`. The -7b' watcher on each node naturally absorbs the rotation. +the rollout with the operations runbook / metrics, then issue +`RotateDEK`. The 7b' watcher on each node naturally absorbs the +rotation. Rollback path: rolling back to 7b after a 7b' RotateDEK has applied is safe (claude review on PR #855 corrected the diff --git a/docs/design/2026_05_28_implemented_7b_runtime_reregistration.md b/docs/design/2026_05_28_implemented_7b_runtime_reregistration.md index 81f852a5f..db8f5c9fc 100644 --- a/docs/design/2026_05_28_implemented_7b_runtime_reregistration.md +++ b/docs/design/2026_05_28_implemented_7b_runtime_reregistration.md @@ -4,7 +4,7 @@ |---|---| | Status | implemented | | Date | 2026-05-28 | -| Parent designs | [`2026_04_29_partial_data_at_rest_encryption.md`](2026_04_29_partial_data_at_rest_encryption.md) (§4.1 writer registry), [`2026_05_26_proposed_7a_process_start_registration.md`](2026_05_26_proposed_7a_process_start_registration.md) (process-start), [`2026_05_26_proposed_7a2_storage_layer_registration_enforcement.md`](2026_05_26_proposed_7a2_storage_layer_registration_enforcement.md) (storage gate) | +| Parent designs | [`2026_04_29_partial_data_at_rest_encryption.md`](2026_04_29_partial_data_at_rest_encryption.md) (§4.1 writer registry), [`2026_05_26_implemented_7a_process_start_registration.md`](2026_05_26_implemented_7a_process_start_registration.md) (process-start), [`2026_05_26_implemented_7a2_storage_layer_registration_enforcement.md`](2026_05_26_implemented_7a2_storage_layer_registration_enforcement.md) (storage gate) | | Builds on | 7a (process-start propose path), 7a-2 (storage-layer `Registered()` gate) | ## 0. Why this slice exists diff --git a/docs/design/2026_05_29_proposed_7c_confchange_time_registration.md b/docs/design/2026_05_29_implemented_7c_confchange_time_registration.md similarity index 95% rename from docs/design/2026_05_29_proposed_7c_confchange_time_registration.md rename to docs/design/2026_05_29_implemented_7c_confchange_time_registration.md index 6384d8223..6e13370f9 100644 --- a/docs/design/2026_05_29_proposed_7c_confchange_time_registration.md +++ b/docs/design/2026_05_29_implemented_7c_confchange_time_registration.md @@ -2,11 +2,20 @@ | Field | Value | |---|---| -| Status | proposed | +| Status | implemented | | Date | 2026-05-29 | -| Parent designs | [`2026_04_29_partial_data_at_rest_encryption.md`](2026_04_29_partial_data_at_rest_encryption.md) (§4.1 writer registry, §5.2 RotateDEK apply), [`2026_05_26_proposed_7a_process_start_registration.md`](2026_05_26_proposed_7a_process_start_registration.md) (§4 forward-looking 7c sketch), [`2026_05_28_implemented_7b_runtime_reregistration.md`](2026_05_28_implemented_7b_runtime_reregistration.md), [`2026_05_28_implemented_7b_prime_runtime_reregistration_rotation.md`](2026_05_28_implemented_7b_prime_runtime_reregistration_rotation.md) | +| Implemented | 2026-07-07 status audit; code landed before this doc promotion | +| Parent designs | [`2026_04_29_partial_data_at_rest_encryption.md`](2026_04_29_partial_data_at_rest_encryption.md) (§4.1 writer registry, §5.2 RotateDEK apply), [`2026_05_26_implemented_7a_process_start_registration.md`](2026_05_26_implemented_7a_process_start_registration.md) (§4 forward-looking 7c sketch), [`2026_05_28_implemented_7b_runtime_reregistration.md`](2026_05_28_implemented_7b_runtime_reregistration.md), [`2026_05_28_implemented_7b_prime_runtime_reregistration_rotation.md`](2026_05_28_implemented_7b_prime_runtime_reregistration_rotation.md) | | Builds on | 7a (process-start propose path), 7a-2 (storage-layer `Registered()` gate), 7b/7b' (runtime watcher) | +## Implementation audit + +Implemented by `internal/raftadmin.MembershipChangeInterceptor`, +`main_encryption_confchange.go`, and `main.go` wiring into raft-admin +servers. +Tests live in `internal/raftadmin/server_test.go` and +`main_encryption_confchange_test.go`. + ## 0. Why this slice exists 7a/7b/7b' cover every registration trigger that fires **on an existing @@ -454,5 +463,8 @@ that bypass the standard workflow) must rely on the `ErrNodeIDCollision` startup membership pre-check before issuing the ConfChange. -This closes Stage 7. Stage 8 (snapshot header v2) and Stage 9 (KMS + -compress + rotation/retire/rewrite + Jepsen) follow. +This closes the 7c ConfChange-time registration slice. The parent +Stage 7 remains partial until the §5.5 `WriterRegistryForCaller` +projection is wired into `GetSidecarState` / `ResyncSidecar`. +Stage 8 (snapshot header v2) and Stage 9 (KMS + compress + +rotation/retire/rewrite + Jepsen) follow. diff --git a/internal/raftadmin/interceptor.go b/internal/raftadmin/interceptor.go index 926f76d39..5cdfe7ce5 100644 --- a/internal/raftadmin/interceptor.go +++ b/internal/raftadmin/interceptor.go @@ -10,7 +10,7 @@ import "context" // // Stage 7c uses this hook to pre-register a new node's writer- // registry row (see -// docs/design/2026_05_29_proposed_7c_confchange_time_registration.md). +// docs/design/2026_05_29_implemented_7c_confchange_time_registration.md). // Keeping the encryption-aware adapter outside this package preserves // raftadmin's engine-generic posture — no concrete dependency on // the KV or encryption layers. diff --git a/main_encryption_confchange.go b/main_encryption_confchange.go index 693cad681..87146f44a 100644 --- a/main_encryption_confchange.go +++ b/main_encryption_confchange.go @@ -25,7 +25,7 @@ import ( // no §4.1 case-4 halt apply on a durable conf-change entry // (guarantee 2: collision-safe membership change). // -// See docs/design/2026_05_29_proposed_7c_confchange_time_registration.md. +// See docs/design/2026_05_29_implemented_7c_confchange_time_registration.md. type encryptionPreRegister struct { coordinate *kv.ShardedCoordinator defaultGroup *kv.ShardGroup