Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/design/2026_04_29_partial_data_at_rest_encryption.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Data-at-rest encryption for elastickv

Status: Partial — Stages 0–5 and Stage 8 shipped (5E deferred); Stage 6 partially shipped; Stages 7 and 9 open
Status: Partial — Stages 0–5 and Stage 8 shipped (5E deferred); Stage 6 partially shipped; Stage 7 partially shipped; Stage 9 open
Author: bootjp
Date: 2026-04-29

Expand Down Expand Up @@ -30,7 +30,7 @@ Date: 2026-04-29
| 6D | §6.6 `enable-storage-envelope` admin RPC + §7.1 Phase-1 storage cutover (§6.2 toggle ON) + Voters ∪ Learners capability gate (depends on 6B for mutator wiring AND 6C-1+6C-2 for §9.1 startup-refusal guards; bundles 6C-3 for the membership-view-dependent collision check — see rationale) | open | — |
| 6E | §6.6 `enable-raft-envelope` admin RPC + §7.1 Phase-2 raft cutover + `raft_envelope_cutover_index` sidecar record + `internal/raftengine/etcd/engine.go` `applyNormalEntry` unwrap hook activation + `ErrRaftUnwrapFailed` HaltApply path + `kv/coordinator.go` / `kv/sharded_coordinator.go` wrap-on-propose switch (Phase-2 leader-side §6.3 proposal-payload wrap) + §7.1 steps 1–6 proposal quiescence barrier (block new user proposal intake, drain in-flight queue, source-tag exemption for the cutover entry itself) + production runtime wiring for startup/apply-time wrap install and post-cutover admin proposals + 6C-4 fail-closed guards. | shipped | 2026_05_31_implemented_6e_enable_raft_envelope.md |
| 6F | §6.5 `--encryption-rotate-on-startup` request flag + leader-elected rotation proposal on the default encryption Raft group. The leader rotates every active DEK purpose once before listeners open; followers keep an in-memory pending request and fire only if they acquire leadership in the same process uptime. | shipped | this PR |
| 7 | Writer registry + deterministic nonce (§4.1) | open | — |
| 7 | Writer registry + deterministic nonce (§4.1). Implemented slices: process-start registration, storage-layer registration gate, runtime re-registration for cutover and rotation, and conf-change-time pre-registration. Still open: §5.5 registry projection in `GetSidecarState` / `ResyncSidecar` (`WriterRegistryForCaller`). | partial | `2026_05_26_implemented_7a_process_start_registration.md` + `2026_05_26_implemented_7a2_storage_layer_registration_enforcement.md` + `2026_05_28_implemented_7b_runtime_reregistration.md` + `2026_05_28_implemented_7b_prime_runtime_reregistration_rotation.md` + `2026_05_29_implemented_7c_confchange_time_registration.md` |
Comment thread
bootjp marked this conversation as resolved.
| 8 | Snapshot header v2 (§4.4); WAL coverage closure (§4.3 / §4.6) | shipped | [`2026_05_29_implemented_8a_snapshot_header_v2.md`](2026_05_29_implemented_8a_snapshot_header_v2.md) + [`2026_06_01_implemented_8b_wal_coverage_closure.md`](2026_06_01_implemented_8b_wal_coverage_closure.md) |
| 9 | KMS-backed wrappers, compression, rotation/retire/rewrite, Jepsen (§5.2, §5.4, §6.4, §8) | open | — |

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,20 @@

| Field | Value |
|---|---|
| Status | proposed |
| Status | implemented |
| Date | 2026-05-26 |
| Parent designs | [`2026_04_29_partial_data_at_rest_encryption.md`](2026_04_29_partial_data_at_rest_encryption.md) (§4.1 writer registry), [`2026_05_26_proposed_7a_process_start_registration.md`](2026_05_26_proposed_7a_process_start_registration.md) (7a coordinator-layer gate) |
| Implemented | 2026-07-07 status audit; code landed before this doc promotion |
| Parent designs | [`2026_04_29_partial_data_at_rest_encryption.md`](2026_04_29_partial_data_at_rest_encryption.md) (§4.1 writer registry), [`2026_05_26_implemented_7a_process_start_registration.md`](2026_05_26_implemented_7a_process_start_registration.md) (7a coordinator-layer gate) |
| Builds on | 7a (coordinator-layer first-write barrier), 6D-6c-1 (`encryption.StateCache`) |

## Implementation audit

Implemented by `internal/encryption.StateCache.Registered`,
`StateCache.MarkRegistered`, `store.WithStorageRegistrationGate`,
`store.ErrWriterNotRegistered`, and the direct-vs-FSM write-path flag in
`store`.
The gate is wired from `main_encryption_write_wiring.go`.

## 0. Why this slice exists

7a's coordinator-layer barrier (`ShardedCoordinator.Dispatch` +
Expand Down Expand Up @@ -270,7 +279,8 @@ forwarder callers before merge.
the first direct-path write does not fail-closed (codex P1).**

### Out of scope
- 7b (post-rotation re-registration), 7c (ConfChange-time registration).
- 7b (post-rotation re-registration), 7c (ConfChange-time registration);
both are implemented by later slices.
- Re-evaluating whether route-catalog data *should* be encrypted at all
(it is, incidentally, when the envelope is active; changing that is a
separate question).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,20 @@

| Field | Value |
|---|---|
| Status | proposed |
| Status | implemented |
| Date | 2026-05-26 |
| Implemented | 2026-07-07 status audit; code landed before this doc promotion |
| Parent design | [`2026_04_29_partial_data_at_rest_encryption.md`](2026_04_29_partial_data_at_rest_encryption.md) (§4.1 writer registry, §5.2 process-start path) |
| Builds on | 6A (`ApplyRegistration` §4.1 case 1–4 dispatch), 5B (`RegisterEncryptionWriter` RPC + leader Proposer wiring), 6C-3 (`ErrLocalEpochRollback`), 6D-6c-2 (`BumpLocalEpoch`, `DeterministicNonceFactory`, keystore hydration) |
| Slice of | Stage 7 (writer registry + deterministic nonce). 7b (post-rotation re-registration) and 7c (ConfChange-time registration) follow. |
| Slice of | Stage 7 (writer registry + deterministic nonce). 7b (post-rotation re-registration) and 7c (ConfChange-time registration) are implemented. |

## Implementation audit

Implemented in `main_encryption_registration.go`, `kv/sharded_coordinator.go`,
and the writer-registry helpers in `store` / `internal/encryption`.
Tests live in `kv/sharded_coordinator_registration_gate_test.go`,
`main_encryption_registration_test.go`, and encryption/store registration
gate suites.

## 0. Why this slice exists

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,20 @@

| Field | Value |
|---|---|
| Status | proposed |
| Status | implemented |
Comment thread
bootjp marked this conversation as resolved.
| Date | 2026-05-28 |
| Parent designs | [`2026_04_29_partial_data_at_rest_encryption.md`](2026_04_29_partial_data_at_rest_encryption.md) (§4.1 writer registry, §5.2 RotateDEK apply), [`2026_05_28_implemented_7b_runtime_reregistration.md`](2026_05_28_implemented_7b_runtime_reregistration.md) (cutover case + §6 deferred-rotation analysis) |
| Builds on | 7a (process-start propose path), 7a-2 (storage-layer `Registered()` gate), 7b (runtime watcher for the cutover case) |
| Implemented | 2026-07-07 status audit; code landed before this doc promotion |

## Implementation audit

Implemented by `internal/encryption.WithLocalEpoch`,
`internal/encryption.WithRaftLocalEpoch`, `writeRotationSidecar`,
`main_encryption_registration.go`'s runtime registration watcher, and
the per-DEK `StateCache.Registered` gate. Tests cover storage and raft
rotation-local-epoch sidecar writes plus runtime rotation registration
scope/verify paths.

## 0. Why this slice exists

Expand Down Expand Up @@ -110,40 +120,38 @@ cleaner but ships materially more code; we will revisit it if option

### 3.1 `applyRotateDEK` per-node sidecar mutation

`applyRotateDEK` currently writes the new key with `LocalEpoch: 0`
via `writeRotationSidecar`. The change:

- Inject a `localEpoch uint16` value into the `Applier` via a new
`WithLocalEpoch(uint16)` functional option (symmetric with
`WithStateCache`, `WithSidecarPath`, etc.).
- The value is `encryptionWriteWiring.epoch` — the same pinned
`w.epoch` the nonce factory is constructed with at process start.
`main_encryption_write_wiring.go`'s `buildEncryptionWriteWiring`
already computes this value; this slice exposes it to the
`Applier` construction site in `main.go`.
- `writeRotationSidecar` reads `a.localEpoch` (defaulting to `0`
when the option was not supplied — preserving the FSM-internal
test harnesses that construct an `Applier` without write-path
state) and writes that value into `sc.Keys[newDEK].LocalEpoch`
**only when `p.Purpose == PurposeStorage`**.

#### 3.1.1 Why the value is storage-only — Raft DEK rotations must keep `LocalEpoch: 0`
Before this slice, `applyRotateDEK` wrote the new key with
`LocalEpoch: 0` via `writeRotationSidecar`. The change:

- Inject purpose-specific epoch values into the `Applier` via
`WithLocalEpoch(uint16)` for storage and
`WithRaftLocalEpoch(uint16)` for raft-envelope writes (symmetric
with `WithStateCache`, `WithSidecarPath`, etc.).
- The values are `encryptionWriteWiring.epoch` and
`encryptionWriteWiring.raftEpoch` — the pinned epochs the storage
and raft-envelope nonce factories are constructed with at process
start. `main_encryption_write_wiring.go`'s
`buildEncryptionWriteWiring` computes both values; this slice
exposes them to the `Applier` construction site in `main.go`.
- `writeRotationSidecar` reads `a.localEpoch` for
`PurposeStorage` and `a.raftLocalEpoch` for `PurposeRaft`
(defaulting to `0` when the option was not supplied — preserving
FSM-internal test harnesses that construct an `Applier` without
write-path state) and writes that value into
`sc.Keys[newDEK].LocalEpoch`.

#### 3.1.1 Why the value is purpose-specific

`applyRotateDEK` handles both `PurposeStorage` and `PurposeRaft`
rotations today (`writeRotationSidecar` dispatches on `p.Purpose`
when assigning `sc.Active.{Storage,Raft}`). The
`encryptionWriteWiring.epoch` value piped in by `WithLocalEpoch` is
the **storage** write-path's pinned `w.epoch` — the value the
deterministic nonce factory under the storage envelope is using.

A raft DEK rotation has its own (future) per-purpose epoch
counter; cross-applying the storage nonce factory's epoch to a
raft DEK's `LocalEpoch` would corrupt the raft DEK's counter
before the raft envelope path consumes it (codex P2 on PR #855).
Until raft envelope support lands with its own per-purpose
plumbing, raft DEK rotations MUST continue to write `LocalEpoch:
0` exactly as today — `writeRotationSidecar`'s switch on
`p.Purpose` gates the new behaviour to storage:
the **storage** write-path's pinned `w.epoch`; the
`encryptionWriteWiring.raftEpoch` value piped in by
`WithRaftLocalEpoch` is the raft-envelope counterpart. Cross-applying
the storage nonce factory's epoch to a raft DEK's `LocalEpoch` would
corrupt the raft counter, so `writeRotationSidecar` selects the
matching epoch for each purpose:

```go
switch p.Purpose {
Expand All @@ -152,7 +160,7 @@ case fsmwire.PurposeStorage:
keyLocalEpoch = a.localEpoch
case fsmwire.PurposeRaft:
sc.Active.Raft = p.DEKID
keyLocalEpoch = 0 // current behaviour preserved
keyLocalEpoch = a.raftLocalEpoch
}
```

Expand Down Expand Up @@ -290,14 +298,15 @@ separate slice can introduce a per-DEK cache.
1. KEK-unwrap new DEK.
2. Keystore set (`a.keystore.Set(p.DEKID, dek)`).
3. **Sidecar write — with `a.localEpoch` for `Keys[newDEK].LocalEpoch`
when `p.Purpose == PurposeStorage`, `0` otherwise** (see §3.1.1).
when `p.Purpose == PurposeStorage`, and `a.raftLocalEpoch` when
`p.Purpose == PurposeRaft`** (see §3.1.1).
4. **Proposer registration** (`ApplyRegistration(p.ProposerRegistration)`)
— proposer's value is `(newDEK, proposer.node_id, proposer.w.epoch)`,
set by the proposer at propose time. Verbatim into the writer
registry.

Step 3 changes from `LocalEpoch: 0` to `LocalEpoch: a.localEpoch` (for
storage). Step 4 is unchanged.
Step 3 changes from `LocalEpoch: 0` to the purpose-specific pinned
epoch. Step 4 is unchanged.

**Sidecar-before-registration is the existing `applyRotateDEK`
ordering** (coderabbitai major review on PR #855 raised concern
Expand Down Expand Up @@ -362,15 +371,21 @@ node's sidecar now records its own emissions accurately.
## 5. Verification action items (for the implementation PR)

1. New `applier_test.go` cases:
- `TestApplier_ApplyRotateDEK_LocalEpochProviderReadAtApply`:
given a `WithLocalEpoch` provider returning 7, the rotation
- `TestApplyRotateDEK_LocalEpochAppliedToStorageKey`:
given `WithLocalEpoch(7)`, the storage rotation
entry under a fresh DEK writes `Keys[newDEK].LocalEpoch = 7`
to the sidecar.
- `TestApplier_ApplyRotateDEK_NoLocalEpochProviderFallsBack`:
- `TestApplyRotateDEK_RaftLocalEpochAppliedToRaftKey`:
given `WithRaftLocalEpoch(9)`, the raft rotation entry writes
`Keys[newDEK].LocalEpoch = 9` to the raft sidecar key.
- `TestApplyRotateDEK_LocalEpoch_RaftRotationKeepsZero`:
constructing the `Applier` without `WithRaftLocalEpoch` keeps
raft rotations at `LocalEpoch: 0` for unwired harnesses.
- `TestApplyRotateDEK_LocalEpoch_NoOptionFallsBackToZero`:
constructing the `Applier` without the option preserves
today's `LocalEpoch: 0` behaviour (test-harness
backward-compatibility).
- `TestApplier_ApplyRotateDEK_LocalEpochAppliedPerApply`:
- `TestApplyRotateDEK_LocalEpoch_AppliedPerApply`:
verify the static `a.localEpoch` value is correctly read
on each FSM apply call, not inadvertently reset or derived
from mutable state. A multi-apply test fixture (two
Expand Down Expand Up @@ -465,18 +480,14 @@ mitigations close this gap, in order of operator friction:
metrics can warn if the cluster reports mixed versions and a
`RotateDEK` is observed in the audit log.

2. **Admin RPC capability gate (middle friction).** The
`EncryptionAdmin.RotateDEK` RPC handler (in the propose-side
mutator) gates on a cluster-wide capability probe — gossiped or
queried via `Distribution.ListNodes` — refusing to propose if
any peer advertises a version older than 7b'. This is
server-side and cannot be bypassed by a misconfigured client.
Failure surface: the RPC returns
`FailedPrecondition: cluster contains pre-7b' nodes; complete
the rolling upgrade before rotating`. The probe is best-effort
(a freshly-joined node could race the check), but the §4.1
case-2 idempotent posture means a false-positive admit only
degrades availability of the pre-7b' node — it never corrupts.
2. **Admin RPC capability gate (middle friction; not shipped in this
slice).** A future `EncryptionAdmin.RotateDEK` RPC handler can gate
on a cluster-wide capability probe — gossiped or queried via
`Distribution.ListNodes` — refusing to propose if any peer
advertises a version older than 7b'. The current handler performs
leader/proposer/input validation but does not fan out this
capability probe, so this slice relies on the operational guideline
in (1) rather than a server-side mixed-version block.

3. **Cluster-version Raft entry (highest friction; deferred).** A
future slice could land a `ClusterVersion` Raft entry that
Expand All @@ -485,13 +496,15 @@ mitigations close this gap, in order of operator friction:
strongest form but adds a new wire entry and a coordination
protocol — overkill for a single-feature gate.

7b' will ship (1) and (2). (3) is explicitly deferred — it has
7b' ships (1). (2) and (3) are explicitly deferred — both have
broader applicability than just this slice (snapshot v2 in Stage 8
will face the same problem) and deserves its own design.
will face the same problem) and deserve their own implementation
designs.

Roll-forward path: deploy 7b' to every node in the cluster, verify
the capability probe reports all-7b', then issue `RotateDEK`. The
7b' watcher on each node naturally absorbs the rotation.
the rollout with the operations runbook / metrics, then issue
`RotateDEK`. The 7b' watcher on each node naturally absorbs the
rotation.

Rollback path: rolling back to 7b after a 7b' RotateDEK has
applied is safe (claude review on PR #855 corrected the
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
|---|---|
| Status | implemented |
| Date | 2026-05-28 |
| Parent designs | [`2026_04_29_partial_data_at_rest_encryption.md`](2026_04_29_partial_data_at_rest_encryption.md) (§4.1 writer registry), [`2026_05_26_proposed_7a_process_start_registration.md`](2026_05_26_proposed_7a_process_start_registration.md) (process-start), [`2026_05_26_proposed_7a2_storage_layer_registration_enforcement.md`](2026_05_26_proposed_7a2_storage_layer_registration_enforcement.md) (storage gate) |
| Parent designs | [`2026_04_29_partial_data_at_rest_encryption.md`](2026_04_29_partial_data_at_rest_encryption.md) (§4.1 writer registry), [`2026_05_26_implemented_7a_process_start_registration.md`](2026_05_26_implemented_7a_process_start_registration.md) (process-start), [`2026_05_26_implemented_7a2_storage_layer_registration_enforcement.md`](2026_05_26_implemented_7a2_storage_layer_registration_enforcement.md) (storage gate) |
| Builds on | 7a (process-start propose path), 7a-2 (storage-layer `Registered()` gate) |

## 0. Why this slice exists
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,20 @@

| Field | Value |
|---|---|
| Status | proposed |
| Status | implemented |
| Date | 2026-05-29 |
| Parent designs | [`2026_04_29_partial_data_at_rest_encryption.md`](2026_04_29_partial_data_at_rest_encryption.md) (§4.1 writer registry, §5.2 RotateDEK apply), [`2026_05_26_proposed_7a_process_start_registration.md`](2026_05_26_proposed_7a_process_start_registration.md) (§4 forward-looking 7c sketch), [`2026_05_28_implemented_7b_runtime_reregistration.md`](2026_05_28_implemented_7b_runtime_reregistration.md), [`2026_05_28_implemented_7b_prime_runtime_reregistration_rotation.md`](2026_05_28_implemented_7b_prime_runtime_reregistration_rotation.md) |
| Implemented | 2026-07-07 status audit; code landed before this doc promotion |
| Parent designs | [`2026_04_29_partial_data_at_rest_encryption.md`](2026_04_29_partial_data_at_rest_encryption.md) (§4.1 writer registry, §5.2 RotateDEK apply), [`2026_05_26_implemented_7a_process_start_registration.md`](2026_05_26_implemented_7a_process_start_registration.md) (§4 forward-looking 7c sketch), [`2026_05_28_implemented_7b_runtime_reregistration.md`](2026_05_28_implemented_7b_runtime_reregistration.md), [`2026_05_28_implemented_7b_prime_runtime_reregistration_rotation.md`](2026_05_28_implemented_7b_prime_runtime_reregistration_rotation.md) |
| Builds on | 7a (process-start propose path), 7a-2 (storage-layer `Registered()` gate), 7b/7b' (runtime watcher) |

## Implementation audit

Implemented by `internal/raftadmin.MembershipChangeInterceptor`,
`main_encryption_confchange.go`, and `main.go` wiring into raft-admin
servers.
Tests live in `internal/raftadmin/server_test.go` and
`main_encryption_confchange_test.go`.

## 0. Why this slice exists

7a/7b/7b' cover every registration trigger that fires **on an existing
Expand Down Expand Up @@ -454,5 +463,8 @@ that bypass the standard workflow) must rely on the
`ErrNodeIDCollision` startup membership pre-check before issuing the
ConfChange.

This closes Stage 7. Stage 8 (snapshot header v2) and Stage 9 (KMS +
compress + rotation/retire/rewrite + Jepsen) follow.
This closes the 7c ConfChange-time registration slice. The parent
Stage 7 remains partial until the §5.5 `WriterRegistryForCaller`
projection is wired into `GetSidecarState` / `ResyncSidecar`.
Stage 8 (snapshot header v2) and Stage 9 (KMS + compress +
rotation/retire/rewrite + Jepsen) follow.
2 changes: 1 addition & 1 deletion internal/raftadmin/interceptor.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import "context"
//
// Stage 7c uses this hook to pre-register a new node's writer-
// registry row (see
// docs/design/2026_05_29_proposed_7c_confchange_time_registration.md).
// docs/design/2026_05_29_implemented_7c_confchange_time_registration.md).
// Keeping the encryption-aware adapter outside this package preserves
// raftadmin's engine-generic posture — no concrete dependency on
// the KV or encryption layers.
Expand Down
2 changes: 1 addition & 1 deletion main_encryption_confchange.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ import (
// no §4.1 case-4 halt apply on a durable conf-change entry
// (guarantee 2: collision-safe membership change).
//
// See docs/design/2026_05_29_proposed_7c_confchange_time_registration.md.
// See docs/design/2026_05_29_implemented_7c_confchange_time_registration.md.
Comment thread
bootjp marked this conversation as resolved.
type encryptionPreRegister struct {
coordinate *kv.ShardedCoordinator
defaultGroup *kv.ShardGroup
Expand Down
Loading