Repository navigation
Repair runfiles trees with entries deleted out-of-band - #46
Merged
Merged
Conversation
If a symlink inside a runfiles tree built by a SymlinkTreeAction is deleted from disk (e.g. by a CI disk cleaner), Bazel never restores it. The output MANIFEST is the action's only output, so the action keeps hitting the action cache, even across server restarts and with --experimental_check_output_files. RunfilesTreeUpdater skips these trees, so any spawn that reads the missing entry fails. On CI the Kotlin builder worker fails to start on every build on the affected host: KotlinCompile ... failed: Worker process did not return a WorkResponse: IllegalStateException: .../build.runfiles/rules_kotlin++rules_kotlin_extensions+kotlin_build_tools_impl/file/downloaded does not exist in the runfiles! Example (Bazel 8.8.0 shadow pipeline; 8.1.1 has the same code): https://buildkite.com/uber/android-code-differential-bazel8/builds/6501#01a10fd3-bd05-4654-a91b-7aa51e5cbc41 Fix: before a local or worker spawn uses a runfiles tree, check once per command that every entry exists, and resync the tree if any are missing. The manifest-digest shortcut for --nobuild_runfile_links trees gets the same check. Testing on 8.1.1: - Reproduced the failure: deleted the kotlin_build_tools_impl symlink from the Kotlin builder's runfiles tree in an Android output base and built cold with --config=buildkite on stock 8.1.1 (1657d0f). The build failed with the error above. - Tested the fix: built the same damaged tree cold with the fixed binary. It logged "Runfiles tree ... is missing entries, recreating it." and the build passed. - RunfilesTreeUpdaterTest: 2 of 3 fail without the fix; 3/3 pass with it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ckilian867
approved these changes
Oct 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
If a symlink inside a runfiles tree built by a SymlinkTreeAction is
deleted from disk (e.g. by a CI disk cleaner), Bazel never restores it.
The output MANIFEST is the action's only output, so the action keeps
hitting the action cache, even across server restarts and with
--experimental_check_output_files. RunfilesTreeUpdater skips these trees,
so any spawn that reads the missing entry fails. On CI the Kotlin builder
worker fails to start on every build on the affected host:
Example (Bazel 8.8.0 shadow pipeline; 8.1.1 has the same code):
https://buildkite.com/uber/android-code-differential-bazel8/builds/6501#01a10fd3-bd05-4654-a91b-7aa51e5cbc41
Fix: before a local or worker spawn uses a runfiles tree, check once per
command that every entry exists, and resync the tree if any are missing.
The manifest-digest shortcut for --nobuild_runfile_links trees gets the
same check.
Testing on 8.1.1:
from the Kotlin builder's runfiles tree in an Android output base and
built cold with --config=buildkite on stock 8.1.1 (1657d0f). The
build failed with the error above.
It logged "Runfiles tree ... is missing entries, recreating it." and
the build passed.
🤖 Generated with Claude Code