Repository navigation
[fence 3/9] libsql-server: fence controller, WAL write gate and positive source drain - #43
Draft
tszymczyszyn-shopify wants to merge 5 commits into
Draft
tszymczyszyn-shopify wants to merge 5 commits into
tszymczyszyn-shopify wants to merge 5 commits into
Conversation
…t connection Add the in-memory authority for namespace fences: - `FenceRegistry`, held by `NamespaceStore` outside the namespace cache and seeded from `MetaStore::load_fences()` before anything is served, so an evicted and reloaded namespace gets the controller it had. Namespaces without fence state get an UNFENCED controller on first load; deleting a namespace drops its controller. - `FenceController`: a per-namespace transition lock and a `watch` gate (`GateSnapshot`: the durable fence, a write generation and an indeterminate flag). Commands commit in the metastore, are published to the gate and only then answered, on their own task so a lost response does not lose the publication. An error before COMMIT leaves the gate unchanged; a failed COMMIT closes the gate and refuses other commands with FENCE_COMMIT_INDETERMINATE until the same command is replayed. - `FenceConnState`, bound to the controller for every connection a `MakeLegacyConnection` opens, starting with its held connection, and shared with the connection's WAL wrapper. The checks that use it land in the next commit. - `cfg(test)` `FenceTestHooks` with the named hook points of the design. `NamespaceStore::with` and `make_namespace` refuse a namespace whose fence state is unavailable before any setup. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
…tion Make ManagedConnectionWalWrapper::begin_write_txn the authoritative namespace fence check. Before queueing for the write slot it requires the live gate to admit the connection's operation class and the program and its read transaction to have been admitted under the gate's current write generation. A refusal returns SQLITE_AUTH (not BUSY, so SQLite does not retry it, and before acquire(), so no slot is released that was never held) and leaves the typed outcome in the connection's FenceConnState. - FenceConnState gains begin_program, begin_read_txn and admit_write. CoreConnection::run, every with_raw call and vacuum_if_needed start a program; the WAL wrapper records the generation of each new read transaction before its snapshot is taken. - The Vm refuses Write and DDL statements early against the live gate and reports a WAL refusal as Error::NamespaceFence instead of SQLITE_AUTH. A plain SQLITE_AUTH from an authorizer is left unchanged. - New detail stale_transaction on MIGRATION_WRITE_FENCED for a transaction or program that began under an earlier generation. - Namespaces without a fence stay at generation 0 and behave as before. Tests cover a program parked between admission and its write while the fence is acquired and released, read-to-write upgrades, DDL, a header pragma, BEGIN IMMEDIATE, VACUUM and raw writes, stale transactions after release, and unchanged behaviour of unfenced namespaces. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
Queue entries and the write slot now carry the operation class. A write transaction waiting for the slot re-checks the connection's fence admission every time it takes the manager's lock, and the fence controller wakes every registered write queue after each change of the write generation, so a writer queued before a fence leaves the queue with MIGRATION_WRITE_FENCED instead of waiting for the slot and then writing. Checkpoints ask for the slot as maintenance: they are never refused and queue again when woken. The manager exposes what the positive write drain needs: the active writer and its class, a notification on every release, and abort_active(), which uses the registered rollback handle. Abort no longer panics when the connection has already closed. VACUUM is skipped, and reported as skipped rather than failed, while the fence denies normal writes, including when the fence closes between the check and the statement. TRUNCATE checkpoints run in every state. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
AcquireSourceWriteFence now runs the drain of docs/NAMESPACE_FENCE.md section 8.3 end to end, under the namespace's transition lock and on a task of its own: - an in-memory INSTALLING gate closes write admission (and moves the write generation, waking queued writers) before SOURCE_DRAINING is persisted; a command proven not to have committed removes it again; - the drain waits on the connection manager's release notification for the writer that held the slot when admission closed, never on elapsed time or the transaction timeout; at the deadline it answers DRAINING (admission stays closed) or, with force_rollback, rolls the writer back and waits for the actual release; - the frozen boundary (log id, last committed frame) is read under the write-slot lock once no writer holds it, and SOURCE_WRITE_FENCED is committed with it; - replaying a DRAINING command resumes the same drain. Primary connection makers register a write-drain source (their connection manager, held weakly, and their replication log) with the namespace's controller; NamespaceStore::execute_fence_command loads the namespace before an acquisition so that the source exists. The default drain deadline is --namespace-fence-default-write-drain-ms (30 s). FrozenBoundary.frame_no becomes optional, for a log without frames. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
…ach boundary Add crash-restart tests for the namespace fence: each server lifetime runs on its own runtime and is ended without any shutdown code while a fence command is parked at a hook point, so the next start takes the real dirty-recovery path on the same directory. They cover every persistence boundary of AcquireSourceWriteFence and ReleaseSourceWriteFence (including a marker that lags the metastore commit), a restart in SOURCE_DRAINING with a writer active at the crash, indeterminate commits through the drain path (applied and not applied), and lost acquisition responses resolved by replay and inspection. The tests exposed that a source restarted while draining could never finish its drain: dirty recovery rebuilds the replication log under a new log id, and completing the drain refused a boundary on a log other than the one the fence was acquired against, leaving the namespace in SOURCE_DRAINING for good. The frozen boundary now names the log that is live when the drain is proven, the record's identity keeps the acquisition log id, and the server warns when the two differ. Write admission was durably closed throughout, so the data at the boundary is unchanged. The BeforeMetastoreCommit test hook can now report a commit as indeterminate without running it. The contract document describes the restart and log rebuild semantics. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
This was referenced Oct 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the per-namespace fence registry and controller, installed before a namespace's first connection so eviction and reload reinstall the same gate.
Gates every write transaction in
ManagedConnectionWalWrapper::begin_write_txnby admission generation, before the writer slot is acquired, so a program or transaction admitted before a fence change cannot write after it. Queued writers are classed and woken to re-check the gate;VACUUMbecomes its own operation class.Implements the positive source write drain (close admission, wait for or force-roll-back the active writer, capture the frozen boundary) and handles indeterminate commits, restart at every persistence boundary, and response loss.
Review focus
This is the piece that changes code on every connection's hot path, fence or no fence: the WAL wrapper, the writer queue and
CoreConnection. It deserves the most careful review and possibly a benchmark.Commits
Stack
Part 3 of 9, based on
namespace-fence/1-model-state. Retargeted from #35 with no feature change: applied in order, the 9 PRs carry #35's fence diff (stable patch ID70d97d6a) onv0.9.30-shopify-patches. Review and land bottom-up, restacking after each squash or rebase merge.