Repository navigation
[fence 4/9] libsql-server: source read fence for SQL, dump and replication streams - #44
Draft
tszymczyszyn-shopify wants to merge 3 commits into
Draft
tszymczyszyn-shopify wants to merge 3 commits into
tszymczyszyn-shopify wants to merge 3 commits into
Conversation
SetSourceReadFence now closes read admission in memory, persists SOURCE_READ_DRAINING, and waits for every read lease already held before it persists SOURCE_READ_FENCED. Leases are taken only after checking the gate under the controller's lease lock, so once admission is closed the set of leases can only shrink. Each SQL program holds a lease for as long as it runs (including a Hrana cursor still producing rows), as does describe and each admin shell query. ATTACH of a namespace is a read of that namespace: the attaching connection keeps a lease on it for every later program until it is detached. A connection idle inside a transaction holds no lease; its next program is refused with MIGRATION_READ_FENCED and rolled back. /beta/listen is refused where reads are denied and ends when reads are fenced. At the deadline (--namespace-fence-default-read-drain-ms, 30s) running programs are cancelled through the connection's progress-handler cancel flag and report the read fence; the drain still waits for the actual releases and answers DRAINING if they do not come, and a replay of the same command resumes it. ClearSourceReadFence reopens reads with writes still fenced. Dump and replication stream leases follow. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
Dump and replication now hold read leases on the namespace fence, so the source read fence drains them positively: - /dump is admitted by the fence gate before a connection is created (a failure to create one is an error, not a panic) and the export holds a dump lease. At the read drain's deadline the export is cancelled before its next row, and a write blocked on a peer that stopped reading fails at once, so the lease is released without the peer. A cancelled dump ends its body with the fence error and never reaches its final COMMIT;. - Replication hello, log_entries, batch_log_entries and snapshot are refused at their start with FAILED_PRECONDITION and x-libsql-fence-code while streams are denied; refusals are counted and logged at most once a minute per namespace. Streams (and the frames of a batch) are served through FencedStream, whose watcher ends the stream as soon as the gate closes or the drain cancels it, dropping the inner stream and the lease itself; the next poll yields the typed terminal status. - With --enable-namespace-fence, the RPC server and the user HTTP server send HTTP/2 keepalive pings (--namespace-fence-keepalive-interval-s, default 30 s, 20 s timeout) so dead peers are detected. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
…mpaction timing `snapshot_stream_ends_typed` polled `get_snapshot_file(1)` and unwrapped its result while the compactor was still writing the first snapshot and the snapshot merger could be replacing merged files. The lookup lists the snapshot directory and then opens the chosen file, so under load it could fail with `NotFound` (directory not created yet, or a file removed by a merge between the listing and the open) and the test panicked. Open the stream through the `snapshot` RPC itself in a bounded loop that treats only "snapshot not found" and the vanished-file error as "not yet", and asserts that no read lease is held after a failed attempt. An opened stream holds its file open, so a later merge cannot affect it. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
This was referenced Oct 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds read leases on the fence controller and the source read fence: new SQL reads are refused with
MIGRATION_READ_FENCED, open reads are drained and cancelled at the deadline. Covers ATTACH (an attached namespace's fence is checked), the admin shell, and/beta/listen.Dump and replication streams hold read leases too: a fenced dump stops before the next row and before its final
COMMIT;, andlog_entries/snapshotstreams end with a typed status. Adds an optional HTTP/2 keepalive so a dead peer's stream can be detected.Review focus
The lease linearisation (gate checked under the lease lock) and the stream cancellation paths.
Commits
Stack
Part 4 of 9, based on
namespace-fence/2-write-gate. Retargeted from #35 with no feature change: applied in order, the 9 PRs carry #35's fence diff (stable patch ID70d97d6a) onv0.9.30-shopify-patches. Review and land bottom-up, restacking after each squash or rebase merge.