Repository navigation
[fence 2/9] libsql-server: fence model, durable state and fail-closed recovery - #42
Closed
tszymczyszyn-shopify wants to merge 5 commits into
Closed
tszymczyszyn-shopify wants to merge 5 commits into
tszymczyszyn-shopify wants to merge 5 commits into
Conversation
This was referenced Oct 5, 2026
tszymczyszyn-shopify
force-pushed
the
namespace-fence/0-contract
branch
from
October 13, 2026 19:41
eef1521 to
d239d38
Compare
tszymczyszyn-shopify
force-pushed
the
namespace-fence/1-model-state
branch
from
October 13, 2026 19:41
872576b to
2e91ab3
Compare
Add the I/O-free core of the namespace fence described in docs/NAMESPACE_FENCE.md: - state.rs: roles, fence states (including the non-durable UNFENCED, ABSENT and derived UNKNOWN_UNAVAILABLE), operation classes and the permission matrix. - outcome.rs: stable outcome codes, their admin HTTP, user HTTP, Hrana, gRPC and proxy mappings (fence denials are never 5xx, 429 or gRPC UNAVAILABLE), bounded detail reasons and FenceError. - command.rs: fence commands and requests, and the canonical SHA-256 request fingerprint over everything except command_id. - record.rs: fence records, command receipts and the on-disk marker, with a strict protobuf encoding (proto/namespace_fence.proto) whose decoder rejects unknown versions, enum values, malformed ids and self-contradictory records instead of defaulting. - transition.rs: the pure apply() and complete_drain() functions. Replay and fingerprint conflicts are decided before owner, role, state and revision; TARGET_WRITABLE has no reverse transition; adoption changes only the owner. Nothing outside the module uses it yet; the metastore, controller and protocol layers follow. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
Add the additive `namespace_fences` and `namespace_fence_receipts` tables (created with --enable-namespace-fence, loaded and enforced whenever they exist) and run every fence command as a compare-and-swap in one BEGIN IMMEDIATE metastore transaction: read the record, the command's receipt and the config row, decide with the pure transition function, then commit the record, the receipt and the legacy block_* mirror together. The revision is stored, so it survives restart. Stored state is read strictly: an unknown format version, an undecodable payload, a revision column that disagrees with its payload, or a per-namespace `.fence` marker that is ahead of the metastore makes the namespace UNKNOWN_UNAVAILABLE instead of guessing. The marker is written after each commit (before it, for target creation) and rewritten on load when it fell behind. Ordinary config writes and deletes now take BEGIN IMMEDIATE and refuse while the fence denies lifecycle operations, and a config write that failed to persist is no longer published to the in-memory config. Receipts of finished operations are pruned after a retention period. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
When namespace fences are in use (the flag is on, the fence tables exist, or a namespace directory holds a fence marker), a namespace whose state startup cannot establish is registered UNKNOWN_UNAVAILABLE instead of being skipped or given a default config: - an undecodable config row, fence row or marker; - a directory with a marker that the metastore has no trustworthy record for: filesystem recovery, a metastore rebuilt by destroy_on_error, a metastore restored from an older backup or without fence tables, and a target whose creation was interrupted. Such a namespace is refused by lookups, config writes and deletes with FENCE_STATE_UNAVAILABLE, is reported by InspectFence, and is settled only by a fence command that commits for it. MetaStore::handle() no longer default-creates on read paths: the new non-creating MetaStore::lookup serves NamespaceStore::with, fork's source and ATTACH authorisation. handle(), used only by creating paths, refuses unavailable names and names without a config whose directory holds a marker. Filesystem recovery skips marked directories, destroy_on_error moves the broken metastore aside instead of deleting it, and a fence row for a namespace name that cannot be decoded stops startup. Servers that never used fences recover as before. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
…efore the commit CreateTargetQuarantined writes the namespace's marker before the transaction that stores its config row, record and receipt. A statement failing in that transaction (SQLITE_FULL, an I/O error) rolled the SQL back but left the marker, so the name stayed UNKNOWN_UNAVAILABLE until the caller replayed the same command, which a caller told of a definite failure has no reason to do. Remove the marker again on any failure before the commit is attempted. A commit that reports a failure keeps it, as does a creation that was already interrupted before.
…ng pool MetaStore::handle read the namespace directory's fence marker with blocking file I/O while it held the configs lock, which every namespace lookup and creation takes. Read it on the blocking pool, before taking the lock.
tszymczyszyn-shopify
force-pushed
the
namespace-fence/1-model-state
branch
from
October 13, 2026 20:51
2e91ab3 to
e87d1d1
Compare
Author
|
Closing. This stack is parked; the note at the top of the description explains why. The branch is kept at |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the
namespace::fencemodule with the fence types, stable outcome codes and the pure transition function (replay and fingerprint checks before owner, role, revision and transition checks), plus thenamespace_fence.protorecord encoding.Persists fences in the metastore: additive tables, a compare-and-swap on a restart-stable revision inside
BEGIN IMMEDIATE, command receipts with retention, the on-disk marker, and the config-write interlock. Makes metastore recovery fail closed when a fence may be involved: non-creatingMetaStore::lookup, a fallibleMetaStore::handle, marker-aware filesystem recovery, anddestroy_on_errorrenaming the metastore aside instead of deleting it.Fence activation and table creation are behind
--enable-namespace-fence(default off), while existing durable fences remain enforced with the flag off. The defensive metastore plumbing and config-write ordering in this piece apply generally, as called out below.Review focus
Metastore changes that apply to everyone, not only fenced namespaces: config writes now use
BEGIN IMMEDIATE, a config whose persist failed is no longer published in memory, andMetaStore::handlecan now fail. The fence module carries a transitional#![allow(dead_code)]until its callers land later in the stack.Since the first version, from review: a definite failure before the commit of
CreateTargetQuarantinedremoves the new target's marker again (so the name is absent, not unavailable), andMetaStore::handlereads the marker of an unknown name on the blocking pool, outside the configs lock.Commits
Stack
Part 2 of 9, based on
namespace-fence/0-contract. The stack sits on #53 (tszymczyszyn/atomic-namespace-create), which sits on #51: review and land #51 → #53 → #41 → … → #49 bottom-up, restacking after each squash or rebase merge.The first version of these PRs carried #35's fence diff unchanged on
v0.9.30-shopify-patches. On 2026-10-13 the stack was rebased onto #53 (how the conflicts were resolved is summarised in a comment on #41) and gained the commits marked above.