Skip to content

Repair runfiles trees with entries deleted out-of-band - #46

Merged
azad-uber-2 merged 1 commit into
uber/android/8.1.1from
azad/repair-runfiles-trees-8.1.1
Oct 6, 2026
Merged

azad-uber-2 merged 1 commit into
uber/android/8.1.1from
azad/repair-runfiles-trees-8.1.1

Conversation

@azad-uber-2

Copy link
Copy Markdown

If a symlink inside a runfiles tree built by a SymlinkTreeAction is
deleted from disk (e.g. by a CI disk cleaner), Bazel never restores it.
The output MANIFEST is the action's only output, so the action keeps
hitting the action cache, even across server restarts and with
--experimental_check_output_files. RunfilesTreeUpdater skips these trees,
so any spawn that reads the missing entry fails. On CI the Kotlin builder
worker fails to start on every build on the affected host:

KotlinCompile ... failed: Worker process did not return a WorkResponse:
IllegalStateException: .../build.runfiles/rules_kotlin++rules_kotlin_extensions+kotlin_build_tools_impl/file/downloaded does not exist in the runfiles!

Example (Bazel 8.8.0 shadow pipeline; 8.1.1 has the same code):
https://buildkite.com/uber/android-code-differential-bazel8/builds/6501#01a10fd3-bd05-4654-a91b-7aa51e5cbc41

Fix: before a local or worker spawn uses a runfiles tree, check once per
command that every entry exists, and resync the tree if any are missing.
The manifest-digest shortcut for --nobuild_runfile_links trees gets the
same check.

Testing on 8.1.1:

  • Reproduced the failure: deleted the kotlin_build_tools_impl symlink
    from the Kotlin builder's runfiles tree in an Android output base and
    built cold with --config=buildkite on stock 8.1.1 (1657d0f). The
    build failed with the error above.
  • Tested the fix: built the same damaged tree cold with the fixed binary.
    It logged "Runfiles tree ... is missing entries, recreating it." and
    the build passed.
  • RunfilesTreeUpdaterTest: 2 of 3 fail without the fix; 3/3 pass with it.

🤖 Generated with Claude Code

If a symlink inside a runfiles tree built by a SymlinkTreeAction is
deleted from disk (e.g. by a CI disk cleaner), Bazel never restores it.
The output MANIFEST is the action's only output, so the action keeps
hitting the action cache, even across server restarts and with
--experimental_check_output_files. RunfilesTreeUpdater skips these trees,
so any spawn that reads the missing entry fails. On CI the Kotlin builder
worker fails to start on every build on the affected host:

  KotlinCompile ... failed: Worker process did not return a WorkResponse:
  IllegalStateException: .../build.runfiles/rules_kotlin++rules_kotlin_extensions+kotlin_build_tools_impl/file/downloaded does not exist in the runfiles!

Example (Bazel 8.8.0 shadow pipeline; 8.1.1 has the same code):
https://buildkite.com/uber/android-code-differential-bazel8/builds/6501#01a10fd3-bd05-4654-a91b-7aa51e5cbc41

Fix: before a local or worker spawn uses a runfiles tree, check once per
command that every entry exists, and resync the tree if any are missing.
The manifest-digest shortcut for --nobuild_runfile_links trees gets the
same check.

Testing on 8.1.1:
- Reproduced the failure: deleted the kotlin_build_tools_impl symlink
  from the Kotlin builder's runfiles tree in an Android output base and
  built cold with --config=buildkite on stock 8.1.1 (1657d0f). The
  build failed with the error above.
- Tested the fix: built the same damaged tree cold with the fixed binary.
  It logged "Runfiles tree ... is missing entries, recreating it." and
  the build passed.
- RunfilesTreeUpdaterTest: 2 of 3 fail without the fix; 3/3 pass with it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@azad-uber-2
azad-uber-2 merged commit 745d544 into uber/android/8.1.1 Oct 6, 2026
3 checks passed
@azad-uber-2
azad-uber-2 deleted the azad/repair-runfiles-trees-8.1.1 branch October 6, 2026 20:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants