Repository navigation
fix(storage): fail fast on out-of-bounds PIP/header page reads - #946
Merged
Merged
Conversation
adsharma
force-pushed
the
fix/843-pip-bounds-validation
branch
from
September 8, 2026 17:51
8907442 to
f7484ab
Compare
adsharma
marked this pull request as ready for review
September 8, 2026 17:51
Corrupted page-index-pointer (PIP) chains previously issued wild preads far beyond EOF (issue #843: position 4727899947008 on a 2884-page file), surfacing as a cryptic IO error on every open, including read-only. Validate page indices against the file size before reading: - FileHandle::readPageFromDisk now throws a RuntimeException naming the offending page and file size (DASSERT alone is stripped in release). - PIPWrapper and DiskArrayCollection header-chain walk check PIP/header indices up front. This converts an unopenable database with a misleading offset into an actionable corruption error, and follows the existing validateCheckpointPageRange pattern in checkpointer.cpp.
adsharma
force-pushed
the
fix/843-pip-bounds-validation
branch
from
September 8, 2026 19:00
f7484ab to
962603d
Compare
This was referenced Sep 9, 2026
fabzter
pushed a commit
to fabzter/ladybug
that referenced
this pull request
Sep 9, 2026
…write path A page index becomes a file offset on the write path exactly as it does on the read path, but only reads validated it (LadybugDB#946). Nothing checked the ranges the allocator handed out, so a corrupted free page list silently extended the data file instead of failing: on the database from LadybugDB#843, PageManager returned page 212,650,604 of a 2884-page file and a checkpoint column flush wrote there, leaving an 11.8 MB database reporting 892.2 GiB of apparent size (LadybugDB#948). Allocated blocks barely move, so nothing looks wrong until a copy or backup materialises the sparse hole, which filled a 926 GiB volume here. Validation at three levels, from the source outwards: - PageManager::allocatePageRange rejects a range that is not inside the data file, naming the range and the file's page count. The free list is deserialized from the database file, so a corrupted entry surfaces here first, and nothing shrinks the data file, so a legitimate free entry is always inside it. The existing DASSERT on the extension path is kept and backed by a real check, since DASSERT is stripped in release builds. - FileHandle::validatePageIdxForWrite guards flushPageIfDirtyWithoutLock and writePagesToFile, which cover the checkpoint column flushes and the spiller, so any other source of a bad index also fails at the write instead of extending the file. - ShadowFile::applyShadowPages and replayShadowPageRecordsCore validate record.originalPageIdx before writing into the data file. Those indices come from deserialized shadow records, and the recovery path runs before any FileHandle exists, so it is bounded by the old header's page count plus the number of shadow pages: a checkpoint can only grow the file by pages it shadowed. Zero-page allocations and zero-length writes are exempt: they write nothing, and their start index legitimately sits at the current end of the file. Verified against the LadybugDB#948 reproducer (a content-scrubbed copy of the LadybugDB#843 database, https://github.com/fabzter/ladybug-948-repro): Page allocation from the free page list returned pages [212650604, 212650605), which are out of bounds for a data file with 2884 pages. The free page list may be corrupted. The file stays 11.8 MB instead of reaching 892 GiB, and the error identifies the corrupted free list rather than surfacing as a SIGSEGV. Checks for false positives: 2000-cycle insert+CHECKPOINT soaks on a clean store carrying FTS and HNSW indexes pass on two seeds; a smoke covering DDL, 3000 inserts with strings up to 3 KB, deletes that free pages, reuse of those freed pages through the free list, and EXPORT/IMPORT all pass. LadybugDB#940 still reproduces, so the guards neither fire spuriously nor mask it.
fabzter
pushed a commit
to fabzter/ladybug
that referenced
this pull request
Sep 9, 2026
A page index becomes a file offset on the write path exactly as it does on the read path, but only reads validated it (LadybugDB#946). Nothing checked the ranges the allocator returned, so a corrupted free page list silently extended the data file instead of failing: on the database from LadybugDB#843, PageManager returned page 212,650,604 of a 2884-page file and a checkpoint column flush wrote there, leaving an 11.8 MB database reporting 892.2 GiB of apparent size (LadybugDB#948). Allocated blocks barely move, so nothing looks wrong until a copy or backup materialises the sparse hole, which filled a 926 GiB volume here. PageManager::allocatePageRange now rejects a range that is not inside the data file, naming the range and the file's page count. The free list is deserialized from the database file, so a corrupted entry surfaces here first, and nothing shrinks the data file, so a legitimate free entry is always inside it. The existing DASSERT on the extension path is kept and backed by a real check, since DASSERT is stripped in release builds. Zero-page allocations are exempt: they write nothing, and their start index legitimately sits at the current end of the file. Verified against the LadybugDB#948 reproducer (a content-scrubbed copy of the LadybugDB#843 database, https://github.com/fabzter/ladybug-948-repro): Page allocation from the free page list returned pages [212650604, 212650605), which are out of bounds for a data file with 2884 pages. The free page list may be corrupted. The file stays 11.8 MB instead of reaching 892 GiB, and the error identifies the corrupted free list rather than surfacing as a SIGSEGV. Checks for false positives: 2000-cycle insert+CHECKPOINT soaks on a clean store carrying FTS and HNSW indexes pass on two seeds, and a smoke covering DDL, 3000 inserts with strings up to 3 KB, deletes that free pages, reuse of those freed pages through the free list, and EXPORT/IMPORT all pass.
fabzter
pushed a commit
to fabzter/ladybug
that referenced
this pull request
Sep 9, 2026
A page index becomes a file offset on the write path exactly as it does on the read path, but only reads validated it (LadybugDB#946). Nothing checked the ranges the allocator returned, so a corrupted free page list silently extended the data file instead of failing: on the database from LadybugDB#843, PageManager returned page 212,650,604 of a 2884-page file and a checkpoint column flush wrote there, leaving an 11.8 MB database reporting 892.2 GiB of apparent size (LadybugDB#948). Allocated blocks barely move, so nothing looks wrong until a copy or backup materialises the sparse hole, which filled a 926 GiB volume here. PageManager::allocatePageRange now rejects a range that is not inside the data file, naming the range and the file's page count. The free list is deserialized from the database file, so a corrupted entry surfaces here first, and nothing shrinks the data file, so a legitimate free entry is always inside it. The existing DASSERT on the extension path is kept and backed by a real check, since DASSERT is stripped in release builds. Zero-page allocations are exempt: they write nothing, and their start index legitimately sits at the current end of the file. Verified against the LadybugDB#948 reproducer (a content-scrubbed copy of the LadybugDB#843 database, https://github.com/fabzter/ladybug-948-repro): Page allocation from the free page list returned pages [212650604, 212650605), which are out of bounds for a data file with 2884 pages. The free page list may be corrupted. The file stays 11.8 MB instead of reaching 892 GiB, and the error identifies the corrupted free list rather than surfacing as a SIGSEGV. Checks for false positives: 2000-cycle insert+CHECKPOINT soaks on a clean store carrying FTS and HNSW indexes pass on two seeds, and a smoke covering DDL, 3000 inserts with strings up to 3 KB, deletes that free pages, reuse of those freed pages through the free list, and EXPORT/IMPORT all pass.
adsharma
pushed a commit
that referenced
this pull request
Sep 9, 2026
A page index becomes a file offset on the write path exactly as it does on the read path, but only reads validated it (#946). Nothing checked the ranges the allocator returned, so a corrupted free page list silently extended the data file instead of failing: on the database from #843, PageManager returned page 212,650,604 of a 2884-page file and a checkpoint column flush wrote there, leaving an 11.8 MB database reporting 892.2 GiB of apparent size (#948). Allocated blocks barely move, so nothing looks wrong until a copy or backup materialises the sparse hole, which filled a 926 GiB volume here. PageManager::allocatePageRange now rejects a range that is not inside the data file, naming the range and the file's page count. The free list is deserialized from the database file, so a corrupted entry surfaces here first, and nothing shrinks the data file, so a legitimate free entry is always inside it. The existing DASSERT on the extension path is kept and backed by a real check, since DASSERT is stripped in release builds. Zero-page allocations are exempt: they write nothing, and their start index legitimately sits at the current end of the file. Verified against the #948 reproducer (a content-scrubbed copy of the #843 database, https://github.com/fabzter/ladybug-948-repro): Page allocation from the free page list returned pages [212650604, 212650605), which are out of bounds for a data file with 2884 pages. The free page list may be corrupted. The file stays 11.8 MB instead of reaching 892 GiB, and the error identifies the corrupted free list rather than surfacing as a SIGSEGV. Checks for false positives: 2000-cycle insert+CHECKPOINT soaks on a clean store carrying FTS and HNSW indexes pass on two seeds, and a smoke covering DDL, 3000 inserts with strings up to 3 KB, deletes that free pages, reuse of those freed pages through the free list, and EXPORT/IMPORT all pass.
adsharma
pushed a commit
that referenced
this pull request
Sep 10, 2026
A page index becomes a file offset on the write path exactly as it does on the read path, but only reads validated it (#946). Nothing checked the ranges the allocator returned, so a corrupted free page list silently extended the data file instead of failing: on the database from #843, PageManager returned page 212,650,604 of a 2884-page file and a checkpoint column flush wrote there, leaving an 11.8 MB database reporting 892.2 GiB of apparent size (#948). Allocated blocks barely move, so nothing looks wrong until a copy or backup materialises the sparse hole, which filled a 926 GiB volume here. PageManager::allocatePageRange now rejects a range that is not inside the data file, naming the range and the file's page count. The free list is deserialized from the database file, so a corrupted entry surfaces here first, and nothing shrinks the data file, so a legitimate free entry is always inside it. The existing DASSERT on the extension path is kept and backed by a real check, since DASSERT is stripped in release builds. Zero-page allocations are exempt: they write nothing, and their start index legitimately sits at the current end of the file. Verified against the #948 reproducer (a content-scrubbed copy of the #843 database, https://github.com/fabzter/ladybug-948-repro): Page allocation from the free page list returned pages [212650604, 212650605), which are out of bounds for a data file with 2884 pages. The free page list may be corrupted. The file stays 11.8 MB instead of reaching 892 GiB, and the error identifies the corrupted free list rather than surfacing as a SIGSEGV. Checks for false positives: 2000-cycle insert+CHECKPOINT soaks on a clean store carrying FTS and HNSW indexes pass on two seeds, and a smoke covering DDL, 3000 inserts with strings up to 3 KB, deletes that free pages, reuse of those freed pages through the free list, and EXPORT/IMPORT all pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related to #843.
A corrupted page-index-pointer (PIP) chain turned into a wild pread far beyond EOF (
position: 4727899947008on a 2884-page file), surfacing as a cryptic IO error on every open, includingread_only.FileHandle::readPageFromDisk: runtime bounds check throwingRuntimeExceptionnaming the offending page and file size (DASSERTalone is stripped in release).PIPWrapperctor andDiskArrayCollectionheader-chain walk: validate PIP/header indices before reading.This converts an unopenable database with a misleading offset into an actionable corruption error, following the existing
validateCheckpointPageRangepattern incheckpointer.cpp. It does not fix the original misdirected write — that writer is still unknown — but it bounds the blast radius and makes the next occurrence diagnosable.