Repository navigation
Keep writer lease renewal from waiting behind fenced writes - #487
Merged
Merged
Conversation
Fixes #486. Every fenced write transaction locked the storage_leases row FOR UPDATE for its whole duration, and renewal is an UPDATE of that same row, so a renewal queued behind any write that was slow or waiting on another lock. Renewal also shared the ten-connection pool, so writers queued on the lease row could leave it no connection, and an attempt had no timeout: one hung renewal blocked every later one until the watchdog stopped the server. Fenced writes now take a per-lease advisory lock, which keeps them serialized as before, and hold the lease row only FOR KEY SHARE. That mode does not conflict with renewal's UPDATE of expires_at, but it does conflict with FOR UPDATE, which acquire now takes explicitly before a takeover. A new holder therefore still waits for every write in flight, and a write that starts after a takeover still sees the new fencing token and fails. Renewal runs on its own connection with a five-second statement timeout, and the adapter abandons a connection that has not answered a second later. main.ts retries a failed renewal every second until the safety deadline instead of exiting on the first error, measures that deadline from when the successful renewal was sent, and logs the last renewal time, how late the watchdog ran, any attempt in flight and the last renewal error when it fires. docs/storage.md describes the lease behavior. Assistant-model: Claude Opus 5.5 (fast) Assistant-workflow: inline (fast subagents) Assistant-verification: red before fix: on 518bd44 the three new renewal tests failed against PostgreSQL 17 (renewal still waiting after 2 s behind a fenced commit and behind a full pool; an attempt blocked on the lease row unsettled after 10 s) Assistant-verification: red before fix: pg_stat_activity showed the renewal UPDATE blocked by a fenced commit that held the lease row while it waited on a channel lock Assistant-verification: takeover safety test passes before and after, and fails when acquire's FOR UPDATE is removed Assistant-verification: PostgreSQL suite passed: 92 pass, 0 fail (apps/server/src/storage/postgres against a disposable PostgreSQL 17) Assistant-verification: bun test passed: 4849 pass, 4 skip, 0 fail (GIT_CONFIG_GLOBAL=/dev/null) Assistant-verification: bun run types passed Assistant-verification: bun run ci passed: dprint, oxlint, tokens, design contract, design record, Impeccable (no new findings) Assistant-verification: local server run: lease row held 18 s, renewal timed out, retried and the server stayed up; held 40 s, watchdog stopped it at 25 s and logged the last renewal and error Co-authored-by: Alex Lavaee <alexlavaee@microsoft.com>
This was referenced Oct 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #486. Every fenced write transaction locked the storage_leases row
FOR UPDATE for its whole duration, and renewal is an UPDATE of that same
row, so a renewal queued behind any write that was slow or waiting on
another lock. Renewal also shared the ten-connection pool, so writers
queued on the lease row could leave it no connection, and an attempt had
no timeout: one hung renewal blocked every later one until the watchdog
stopped the server.
Fenced writes now take a per-lease advisory lock, which keeps them
serialized as before, and hold the lease row only FOR KEY SHARE. That
mode does not conflict with renewal's UPDATE of expires_at, but it does
conflict with FOR UPDATE, which acquire now takes explicitly before a
takeover. A new holder therefore still waits for every write in flight,
and a write that starts after a takeover still sees the new fencing
token and fails.
Renewal runs on its own connection with a five-second statement
timeout, and the adapter abandons a connection that has not answered a
second later. main.ts retries a failed renewal every second until the
safety deadline instead of exiting on the first error, measures that
deadline from when the successful renewal was sent, and logs the last
renewal time, how late the watchdog ran, any attempt in flight and the
last renewal error when it fires. docs/storage.md describes the lease
behavior.
Assistant-model: Claude Opus 5.5 (fast)
Assistant-workflow: inline (fast subagents)
Assistant-verification: red before fix: on 518bd44 the three new renewal tests failed against PostgreSQL 17 (renewal still waiting after 2 s behind a fenced commit and behind a full pool; an attempt blocked on the lease row unsettled after 10 s)
Assistant-verification: red before fix: pg_stat_activity showed the renewal UPDATE blocked by a fenced commit that held the lease row while it waited on a channel lock
Assistant-verification: takeover safety test passes before and after, and fails when acquire's FOR UPDATE is removed
Assistant-verification: PostgreSQL suite passed: 92 pass, 0 fail (apps/server/src/storage/postgres against a disposable PostgreSQL 17)
Assistant-verification: bun test passed: 4849 pass, 4 skip, 0 fail (GIT_CONFIG_GLOBAL=/dev/null)
Assistant-verification: bun run types passed
Assistant-verification: bun run ci passed: dprint, oxlint, tokens, design contract, design record, Impeccable (no new findings)
Assistant-verification: local server run: lease row held 18 s, renewal timed out, retried and the server stayed up; held 40 s, watchdog stopped it at 25 s and logged the last renewal and error
Co-authored-by: Alex Lavaee alexlavaee@microsoft.com
Fixes #486
Assistant-verification: Sol and Luna fast review passed: fencing interleavings against a disposable PostgreSQL 17, no P0/P1 (P2 follow-ups: renewal can extend a just-expired lease after a lock wait without a second writer; tidy watchdog/leaseRetry cleanup on drain and startup failure)