Support incremental VFS dumps into a reused build directory, for CVLR rust builds - #39
Merged
Merged
Conversation
`_LayeredMaterializer` fills a fresh directory: every dump writes every file. That is right for a temp dir, which is the only target `fs_tools_layered` has had, and it becomes wrong as soon as something else owns state in the target — which is what happens the moment a build system runs there. Two problems appear together, and neither is visible with a temp dir: An unchanged file must keep its mtime. A build system that fingerprints on mtime — cargo does — treats a rewritten-but-identical file as a change and rebuilds everything downstream of it. Over a dependency graph that is minutes per dump, which is the entire cost a warm directory exists to avoid. And nothing the materializer did not write may be disturbed. A persistent build directory accumulates compiler output, a package cache and lock files. None of it is view content, and a dump that cleaned the directory would throw away exactly what reusing it was for. Both follow from writing through a manifest kept in the target: compare before writing, and delete only paths an earlier dump of this materializer wrote that the view no longer serves. The manifest lives in the target rather than in memory because the point of a persistent target is that it outlives the process that filled it. Binary content is carried by the first dump, which delegates to each backend's own `dump_to`; incremental dumps go through `get`, which is text-only. Nothing is lost by that — an edit layer's content is `str`, so no edit can change a binary file in the first place. Selected by passing `persistent_materializer` as `fs_tools_layered`'s new `materializer` argument: a factory rather than a flag, so a caller names a strategy and a third one needs no further boolean.
Wiring the first real consumer up found two things a flat backend stack cannot express, and one it expresses at a cost the class exists to avoid. The cost first. A project checkout is the bulk of a view and none of its churn, so content-comparing all of it on every dump is the copy this class was written to avoid, moved rather than removed. `base` is dumped once into a fresh target by the backend's own `dump_to` and never read again; `overlays` are the layers that move and are compared every time. That split then answers a question a flat stack could only get wrong: what happens to a path an overlay stops serving. Deleting it is right when the overlay invented the file and wrong when the overlay *modified* one the project ships — deleting that breaks a build that was fine before anyone edited anything. With a base to consult the rule states itself: restore from the base if it serves the path, remove only if nothing does. Both are what an undo should mean, and which one applies is not something the overlay knows. `persistent_materializer` reads a read-stack the same way round: the last backend is the base, since a read stack lists highest priority first.
An `{path: content}` map is the canonical thing to put above a directory, and
every consumer of the persistent materializer so far has had to write its own —
the tests here did, and so did the first real caller. Mutable on purpose: an
overlay's whole job is to change between dumps, and a caller deriving its
content from somewhere else reassigns `files` and materializes again.
The class docstring was a design essay. Replace it with the contract — reused target, one-shot base copy, overlay compare, restore-vs-unlink — and note that binary files cannot be restored because get is text-only. Trim the surrounding comments the same way.
ericeil
changed the title
Commit the manifest with tmp+replace so a killed dump cannot empty it.
Treat well-formed non-object JSON as absent. Remove leftover {name}.tmp-*
scratch on the next dump of that path, including when the file is unchanged.
ericeil
changed the title
ericeil
marked this pull request as ready for review
September 14, 2026 19:18
jtoman
approved these changes
Sep 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AutoProver's CVLR backend will verify Solana programs in one reused cargo tree. The agent reads a layered VFS (checkout underneath, authored and edited files on top); cargo has to compile that same view. The default materializer fills a fresh directory and rewrites every file, which is wrong here: cargo fingerprints on mtime, and the tree also holds
target/and a privateCARGO_HOMEthat a dump must not touch.PersistentMaterializeris the dump strategy for that tree. The checkout is copied once; overlays are content-compared on every dump; a dropped overlay path is restored from the base or unlinked (revert_munge).DictBackendis the overlay. Passmaterializer=persistent_materializertofs_tools_layered; the default is unchanged.Read tools and materializer still come from one stack, so the agent and the compiler cannot disagree about the view.