Skip to content

Fix use-after-free in async host ledger reads on shutdown#8003

Open
achamayou with Copilot wants to merge 22 commits into
mainfrom
copilot/fix-potential-async-cleanup-problems
Open

Fix use-after-free in async host ledger reads on shutdown#8003
achamayou with Copilot wants to merge 22 commits into
mainfrom
copilot/fix-potential-async-cleanup-problems

Conversation

Copilot AI commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Fix use-after-free in async host ledger reads on shutdown

Problem

When the host Ledger object is destroyed during shutdown, any in-flight asynchronous ledger read workers (dispatched via uv_queue_work) could still be running or about to start on the libuv thread pool. These workers hold a raw Ledger* pointer and access ledger state — leading to a potential use-after-free.

Solution

Introduce a WorkerShutdownGate class (src/ds/worker_shutdown_gate.h) — a reusable cooperative shutdown primitive that:

  1. Workers call try_register() before accessing the protected resource. Returns false if shutdown has begun (worker must skip).
  2. Workers call unregister() when done. Notifies the shutdown waiter.
  3. The owner calls shutdown_and_wait() in the destructor — sets the "shutting down" flag and blocks until all registered workers finish.

The Ledger destructor now calls shutdown_gate->shutdown_and_wait(), ensuring:

  • Workers already accessing Ledger state complete safely before destruction
  • Workers that haven't started yet see the shutdown flag and skip the read

Design

The shutdown protocol is extracted as a standalone, independently testable class rather than being baked into Ledger's internals. This means:

  • No friend structs or test hooks in production code
  • The gate's API (try_register/unregister/shutdown_and_wait) is its real API, not test scaffolding
  • Unit tests for the concurrency protocol use synthetic threads — deterministic and fast
  • The ledger integration test exercises the real ringbuffer dispatch path

Changes

  • src/ds/worker_shutdown_gate.h — New standalone concurrency primitive
  • src/ds/test/worker_shutdown_gate_test.cpp — Deterministic unit tests for the gate
  • src/host/ledger.h — Replace hand-rolled AsyncReadState with WorkerShutdownGate; remove friend struct LedgerAsyncTestAccess and test_worker_active_hook
  • src/host/test/ledger.cpp — Replace friend-based tests with a stress test using the real ringbuffer message path
  • CMakeLists.txt — Add worker_shutdown_gate_test target; version bump
  • CHANGELOG.md — Document the fix

Closes #8003

Copilot AI and others added 2 commits June 29, 2026 17:32
Closes #3501

Co-authored-by: achamayou <4016369+achamayou@users.noreply.github.com>
Closes #3501

Co-authored-by: achamayou <4016369+achamayou@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix potential async cleanup problems Fix use-after-free in async host ledger reads on shutdown Jun 29, 2026
Copilot AI requested a review from achamayou June 29, 2026 17:35
@achamayou
achamayou requested a review from Copilot June 29, 2026 20:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses a shutdown-only use-after-free risk in the host’s async ledger read path (uv_work_t threadpool jobs) by ensuring worker callbacks never dereference a Ledger* after the owning asynchost::Ledger has been destroyed. It does this by introducing a shared control block that outlives the Ledger and by having ~Ledger() wait for in-flight worker callbacks to finish before allowing member destruction.

Changes:

  • Added a shared_ptr-owned AsyncReadState (mutex + condition variable + active_workers + ledger_alive) shared between Ledger and each AsyncLedgerGet.
  • Updated on_ledger_get_async to (a) skip reads once shutdown begins and (b) register/deregister active worker callbacks so ~Ledger() can safely drain them.
  • Added a CHANGELOG entry documenting the shutdown use-after-free fix.

Custom instructions used:

  • .github/copilot-instructions.md
  • .github/instructions/reviewing.instructions.md

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
src/host/ledger.h Introduces AsyncReadState and shutdown coordination so async libuv workers don’t access a destroyed Ledger during/after shutdown.
CHANGELOG.md Documents the shutdown UAF fix with a PR reference.

Copy link
Copy Markdown
Member

No blocking issues found.

I reviewed PR #8003 against main, focusing on src/host/ledger.h async uv_work_t reads, the run_main_loop shutdown order, the linked #3501 concern, minimality, and the repo review instructions.

Correctness:

  • The bug is real: uv_stop in handle_ring_buffer.h exits uv_run without draining threadpool work; Ledger is stack-local in run_main_loop and is destroyed before the post-loop uv_loop_close drain. A pending or running on_ledger_get_async could previously dereference data->ledger after destruction.
  • The AsyncReadState protocol is sound: ledger_alive and active_workers are checked/updated under one mutex, so a worker either registers before the destructor can finish or observes shutdown and never touches Ledger. condition_variable::wait releases the mutex, and the decrement happens in the worker callback, so this does not depend on the stopped main loop running.
  • Completion callbacks remain Ledger-independent: result_cb only uses the captured WriterPtr and request data, and AsyncLedgerGet/uv_work_t are still deleted in completion. The captured writer chain/circuit outlives the loop-close drain.
  • I did not find lock-order inversion with state_lock/read_cache_lock; async_state->lock is not held during read_entries_range.
  • The change is local and minimal for the problem. uv_cancel alone would not cover running work, and moving the loop-close drain earlier would have broader shutdown implications.
  • The CHANGELOG.md entry is appropriate under the current Fixed section.

Non-blocking suggestions:

  • Consider making the active_workers decrement exception-safe with a small scope guard around read_entries_range (src/host/ledger.h:1763-1774). An exception escaping the libuv worker would already be fatal today, so I would not block on this, but the new destructor invariant would be more self-contained if every exit path decremented and notified.
  • The comments around AsyncReadState slightly overstate its role in making completion callbacks safe. It protects the worker's access to Ledger; completion is safe because it no longer touches Ledger and captures to_enclave by value. Tightening that wording would reduce future confusion.
  • I do not think a deterministic test is required here. The failure is a shutdown race around libuv worker scheduling; an ASAN stress test could add confidence but would likely be timing-sensitive.

CI note: At the time of review, the main VM/ACI checks for the current head were still queued or in progress; ASAN/TSAN and long jobs were skipped by workflow configuration.

Custom instructions used:

  • .github/copilot-instructions.md
  • .github/instructions/reviewing.instructions.md

Copilot AI and others added 2 commits June 30, 2026 21:05
Make the active worker accounting exception-safe and clarify that AsyncReadState protects worker access to Ledger while completion callbacks are Ledger-independent.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@achamayou achamayou added the run-long-test Run Long Test job label Jul 3, 2026
@achamayou
achamayou marked this pull request as ready for review July 3, 2026 13:56
@achamayou
achamayou requested a review from a team as a code owner July 3, 2026 13:56
Adds a ledger_test case that queues an in-flight async committed-range read, holds it active on a libuv threadpool thread, then destroys the Ledger on another thread and asserts ~Ledger blocks until the worker completes - exercising the use-after-free fix from #8003. Validated under ASAN/UBSan.

- ledger.h: add guarded (TEST_MODE_LEDGER_ASYNC_HOOK) worker hook and test_queue_async_read trigger; tidy async_read_state comment

- ledger_test: new test case; link target against libuv

- CHANGELOG: move the #8003 entry to the current 7.0.7 section

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Comment thread src/host/ledger.h Outdated
achamayou and others added 3 commits July 3, 2026 15:40
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Replace the manual hook/thread teardown with a single RAII guard that, on every exit path (including a failed REQUIRE, which doctest throws), releases the in-flight worker so ~Ledger can finish, joins the destroyer thread (a joinable std::thread must never be destroyed), drains the completion callback, and clears the global worker hook so it can never retain dangling references to the test frame.
test_queue_async_read now takes an optional result callback so a test can observe the read outcome. Adds a test that queues a read after ~Ledger has begun (ledger_alive == false) and asserts it is skipped rather than served, and extracts the shutdown cleanup into a shared helper reused by both async shutdown tests.
Comment thread src/host/ledger.h Outdated
achamayou and others added 2 commits July 6, 2026 09:32
Address review feedback on PR #8003: avoid putting test-only code
behind an #ifdef in the production header. Replace
TEST_MODE_LEDGER_ASYNC_HOOK (a global mutable hook function and a
public test_queue_async_read method) with:

- An always-compiled, no-op-by-default test_worker_active_hook field
  on the private AsyncReadState struct, called unconditionally in
  on_ledger_get_async.
- A forward-declared LedgerAsyncTestAccess struct, befriended by
  Ledger, defined only in src/host/test/ledger.cpp, which uses that
  friendship to reach the private async_read_state member and queue
  async reads via the already-public AsyncLedgerGet,
  on_ledger_get_async and on_ledger_get_async_complete.

No behaviour change in production builds; the two async shutdown
tests are updated to use the new accessor.
Comment thread src/host/ledger.h Outdated
Comment thread src/host/ledger.h Outdated
achamayou and others added 2 commits July 14, 2026 14:54
@achamayou
achamayou requested a review from maxtropets July 14, 2026 15:03
Comment thread CHANGELOG.md Outdated
Co-authored-by: cjen1-msft <190360281+cjen1-msft@users.noreply.github.com>
Copilot AI requested a review from cjen1-msft July 20, 2026 17:35
eddyashton and others added 2 commits July 21, 2026 12:58
Replace the LedgerAsyncTestAccess friend struct and test_worker_active_hook
with a standalone WorkerShutdownGate class in src/ds/worker_shutdown_gate.h.

The shutdown gate is a reusable concurrency primitive that:
- Allows workers to register/unregister
- Blocks shutdown until all active workers complete
- Rejects new registrations after shutdown begins

This removes all test-only surface from the production Ledger class:
- No friend struct
- No test hook field in production code
- No manual job construction in tests

The ledger test now uses the real ringbuffer dispatch path to trigger async
reads, and exercises concurrent destruction as a stress test (best run
under TSAN/ASAN). The shutdown protocol itself is unit-tested independently
via deterministic thread-based tests in worker_shutdown_gate_test.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants