Repository navigation
[fence 9/9] libsql-server: fence operations: provenance, adoption, metrics - #49
Draft
tszymczyszyn-shopify wants to merge 5 commits into
Draft
tszymczyszyn-shopify wants to merge 5 commits into
tszymczyszyn-shopify wants to merge 5 commits into
Conversation
Keep what the metastore's bottomless restore reports at startup (whether it recovered the database and the generation it restored from) instead of discarding it, record it on the MetaStore, and report it in the fence capability endpoint (`metastore`), in every fence view (`provenance`), in the `libsql_server_metastore_restored_from_backup` gauge and in a startup warning. When destroy_on_error rebuilds the metastore, the restore of the rebuilt metastore is what is reported. Tests cover a real bottomless restore against a local S3 endpoint, the admin rendering, and that `incarnation.current_log_id` names the rebuilt replication log after a dirty restart while the stored record keeps the log the source was acquired on. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
Serve AdoptFence on the admin API (POST /v1/namespaces/:ns/fence/adopt). It needs the admin credential and the separate adoption key configured with --namespace-fence-adoption-key (SQLD_NAMESPACE_FENCE_ADOPTION_KEY), presented in the x-libsql-fence-adoption-key header. The key is kept as a SHA-256 digest and compared without an early exit; without a key, adoption is refused with adoption_not_authorised. The request names the current owner, two distinct approvers, an incident reference and a reason, which are stored in the record and receipt and written to one audit event (target libsql_server::fence::audit). Adoption moves the owner and the revision and nothing else: every admission stays as it was, capabilities of the old owner are revoked, and finished operations cannot be adopted. After a metastore rollback it re-establishes the record the marker holds under the new owner, settles the unavailable name and restores the namespace's own block_* values in memory. When the metastore holds no config row for the name, adoption is refused with namespace_config_missing instead of inventing a configuration. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
Every fence command the server answers (committed, replayed or refused) now emits one structured event under the libsql_server::fence::audit tracing target, with the namespace, operation and command id, outcome, revisions, state before and after, drain kind and duration, forced actions, replay/conflict and server instance; a committed adoption keeps its approvers, incident reference and reason. Metrics, all with bounded labels (docs/NAMESPACE_FENCE.md section 15): transitions by command and outcome, drain duration by kind, forced rollbacks and cancellations by kind, replays and conflicts, denials by code and surface (http, hrana, rpc, proxy, dump, replication, admin_shell, lifecycle), adoptions, and two gauges computed from the fence registry when /metrics is read: namespaces by role and state, and the age of the oldest active fence. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
…dule The fence module carried a module-wide `allow(dead_code)` while its consumers landed across the series. Everything is wired now, so remove it and deal with the five items it was hiding: - delete the `InBeginWriteTxnAfterCheck` and `AfterManagerRelease` hook points, which no code path reaches and no test arms; - delete the unused `AuditReport::drain` accessor and the `FenceController::cancel_read_leases` wrapper (its one test now uses `cancel_read_leases_by_kind`); - compile `DenialSurface::ALL`, `AuditReport::forced_kinds` and the `Fail`/`Indeterminate` hook outcomes only in the library's test build, which is the only place they are used or produced. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
This was referenced Oct 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Surfaces metastore restore provenance and the live log id in fence inspection, adds audited incident adoption (separate adoption key plus two recorded approvers), restart and eviction integration tests over the public protocol, and fence metrics plus a structured audit log (namespace and operation IDs only in logs, never metric labels). Removes the transitional dead-code allowances.
Review focus
Adoption's separate key and two distinct recorded approvers, plus metric label hygiene. The integration tests live here because their harness uses the adoption key.
Commits
Stack
Part 9 of 9, based on
namespace-fence/7-replicas. Retargeted from #35 with no feature change: applied in order, the 9 PRs carry #35's fence diff (stable patch ID70d97d6a) onv0.9.30-shopify-patches. Review and land bottom-up, restacking after each squash or rebase merge.