Skip to content

fix(storage): validate allocated page ranges and page indices on the write path - #950

Merged
adsharma merged 1 commit into
LadybugDB:mainfrom
fabzter:fix/948-write-path-page-bounds
Sep 9, 2026
Merged

adsharma merged 1 commit into
LadybugDB:mainfrom
fabzter:fix/948-write-path-page-bounds

Conversation

@fabzter

@fabzter fabzter commented Sep 9, 2026

Copy link
Copy Markdown

Fixes #948.

Symptom

A single bad page index extends the data file far past its end. On the database from #843 an
11.8 MB file ends up reporting 892.2 GiB of apparent size while only ~12 MiB is allocated.
The database still opens and queries fine, so nothing looks wrong — until something copies or
archives it and the sparse hole is materialised. That is how I found it: one shutil.copy2()
filled a 926 GiB volume to 100% (523 MiB free) before failing with ENOSPC. A backup tool,
tar, or a container build would do the same.

Reproduced identically on 0.19.1, 0.20.2 and main, always the same byte count.

Root cause

A page index becomes a file offset on the write path exactly as it does on the read path, but
only reads were validated (#946). Nothing checked the ranges the allocator handed out.

I traced the actual write with a backtrace on an instrumented build. It is not the shadow-replay
path I first suspected:

LocalFileSystem::writeFile(offset=957961232384)
  <- CompressedFlushBuffer::operator()(span<const uint8_t>, FileHandle*, const PageRange&, const ColumnChunkMetadata&)
  <- ColumnChunkData::flush(PageAllocator&)
  <- Column::checkpointColumnChunkOutOfPlace(...)  /  StringColumn::checkpointSegment(...)
  <- ColumnChunk::checkpoint(...) <- NodeGroup::checkpointInMemAndOnDisk(...)
  <- NodeTable::checkpoint(...) <- StorageManager::checkpoint(...) <- Checkpointer::checkpointStorage()

The range came from PageManager::allocatePageRange, which returns whatever
FreeSpaceManager::popFreePages yields. That free list is deserialized from the database file, so
a corrupted entry is handed out as if it were real: on this database, page 212,650,604 of a
2884-page file. The checkpoint flush then writes there and the file grows to 892 GiB.

This also closes a gap in #843: the guard's message proves that database's free page list is
corrupted, which is the missing link between the page corruption reported there and the file
inflation reported in #948.

Fix

PageManager::allocatePageRange rejects a range that is not inside the data file, naming the
range and the file's page count. This is where a corrupted free list surfaces first, and nothing
shrinks the data file, so a legitimate free entry is always inside it. The existing DASSERT on
the extension path is kept and backed by a real check, since DASSERT is stripped in release
builds. Zero-page allocations are exempt — they write nothing, and their start index legitimately
sits at the current end of the file. Style and message shape follow #946.

An earlier revision of this PR also guarded the FileHandle write paths and both shadow-replay
writes as defense in depth. I removed those, because your CI showed the recovery-path bound
was simply wrong: FlakyCheckpointerTest.RecoverFromCheckpointApplyingShadowFailure legitimately
replays a shadow page into data page 266 of a file that currently has 265 pages — extending the
data file is exactly what recovery is for, so there is no sound page-count bound available at
that point. I could not prove the spiller's use of writePagesToFile safe either. The remaining
change is the one place with an authoritative bound, and it is the place the fault actually
originates.

Verification

On the #948 reproducer (a content-scrubbed copy of the #843 database, published at
https://github.com/fabzter/ladybug-948-repro so this is reproducible without my private data):

Runtime exception: Page allocation from the free page list returned pages
[212650604, 212650605), which are out of bounds for a data file with 2884 pages.
The free page list may be corrupted.

The file stays 11.8 MB instead of reaching 892 GiB, and the error identifies the corrupted free
list instead of surfacing as a SIGSEGV.

False-positive checks (an earlier revision of this patch rejected zero-page allocations, which the
first of these caught):

  • 2000-cycle insert + CHECKPOINT soak on a clean store carrying FTS and HNSW indexes, seeds 1
    and 2: both complete, row counts correct.
  • Smoke covering DDL, 3000 inserts with strings up to 3 KB, deletes that free pages, reuse of
    those freed pages through the free list
    (the path this patch guards), and EXPORT/IMPORT:
    all pass.

I have not run the C++ suite locally — BUILD_TESTS was off in my build directory — so CI is the
first full run. I did not add a unit test because provoking this needs a database whose free list
is already corrupt; happy to add a direct unit test of the guards if you would like one.

Relationship to the other open reports

@fabzter

fabzter commented Sep 9, 2026

Copy link
Copy Markdown
Author

Updated: force-pushed a reduced version after your CI caught a genuine defect in my first
revision. Thanks, that is exactly the failure I could not have found locally.

FlakyCheckpointerTest.RecoverFromCheckpointApplyingShadowFailure failed with my exception:

Cannot replay shadow page into data page 266 during recovery replay:
out of bounds for a data file of 265 pages. The shadow file may be corrupted.

That test is right and I was wrong: recovery replaying a checkpoint legitimately writes past the
current end of the data file, since extending it is the point. My bound there (the old header's
page count plus the shadow page count) has no sound basis, so I have dropped the shadow-replay
guards and the FileHandle write guards entirely — I could not prove the spiller's
writePagesToFile path safe either.

What is left is the single guard in PageManager::allocatePageRange, which is where the fault
actually originates and the one place with an authoritative bound. It still catches #948 with the
same message and the file stays 11.8 MB. Re-verified after the reduction: the reproducer stops
cleanly, 2000-cycle insert+CHECKPOINT soaks with FTS and HNSW indexes pass on two seeds, and the
delete / free-page-reuse / EXPORT+IMPORT smoke passes.

One other thing from that run, for the record rather than as a claim about my change:
e2e_test_ddl~partitioned.PartitionedAlterRestrictions also failed, with Fatal signal 11 right
after MATCH (n:Q) RETURN n.id, n.k on a renamed partitioned table. I do not think it is mine —
the patch only adds a comparison that throws — but I cannot rule it out from here, and it did not
appear in the runs I checked on main or on #949. If it reappears on this run with the reduced
patch I will dig into it properly.

@fabzter

fabzter commented Sep 9, 2026

Copy link
Copy Markdown
Author

@adsharma for some reason I cannot add you as a reviewer but please serve yourself to check and destroy my PR with your knowledge.

Comment thread src/storage/page_manager.cpp Outdated
Comment thread src/storage/page_manager.cpp Outdated
@adsharma

adsharma commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Thank you for the contribution! The code looks correct to me. Minor nits - please fix.

A page index becomes a file offset on the write path exactly as it does on the
read path, but only reads validated it (LadybugDB#946). Nothing checked the ranges the
allocator returned, so a corrupted free page list silently extended the data
file instead of failing: on the database from LadybugDB#843, PageManager returned page
212,650,604 of a 2884-page file and a checkpoint column flush wrote there,
leaving an 11.8 MB database reporting 892.2 GiB of apparent size (LadybugDB#948).
Allocated blocks barely move, so nothing looks wrong until a copy or backup
materialises the sparse hole, which filled a 926 GiB volume here.

PageManager::allocatePageRange now rejects a range that is not inside the data
file, naming the range and the file's page count. The free list is deserialized
from the database file, so a corrupted entry surfaces here first, and nothing
shrinks the data file, so a legitimate free entry is always inside it. The
existing DASSERT on the extension path is kept and backed by a real check,
since DASSERT is stripped in release builds. Zero-page allocations are exempt:
they write nothing, and their start index legitimately sits at the current end
of the file.

Verified against the LadybugDB#948 reproducer (a content-scrubbed copy of the LadybugDB#843
database, https://github.com/fabzter/ladybug-948-repro):

  Page allocation from the free page list returned pages
  [212650604, 212650605), which are out of bounds for a data file with 2884
  pages. The free page list may be corrupted.

The file stays 11.8 MB instead of reaching 892 GiB, and the error identifies
the corrupted free list rather than surfacing as a SIGSEGV. Checks for false
positives: 2000-cycle insert+CHECKPOINT soaks on a clean store carrying FTS and
HNSW indexes pass on two seeds, and a smoke covering DDL, 3000 inserts with
strings up to 3 KB, deletes that free pages, reuse of those freed pages through
the free list, and EXPORT/IMPORT all pass.
@fabzter
fabzter force-pushed the fix/948-write-path-page-bounds branch from 3b07a94 to f11501d Compare September 9, 2026 15:26
@adsharma
adsharma merged commit db7830a into LadybugDB:main Sep 9, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants