Skip to content

Settle futures on any exception; validate WorkQueue/Progress input - #628

Draft
GoodForOneFare wants to merge 1 commit into
mainfrom
gordo-work-queue-progress-guards
Draft

Settle futures on any exception; validate WorkQueue/Progress input#628
GoodForOneFare wants to merge 1 commit into
mainfrom
gordo-work-queue-progress-guards

Conversation

@GoodForOneFare

@GoodForOneFare GoodForOneFare commented Aug 14, 2026

Copy link
Copy Markdown
Member

The main bug: non-StandardError exceptions hang SpinGroup forever

WorkQueue's worker rescued only StandardError and Interrupt. Any other exception escaped the worker thread without completing its future:

  • The future stays incomplete forever, so Future#value blocks indefinitely.
  • The dead worker thread stays in the workers array and keeps counting against max_concurrent, so queued work never gets a replacement worker.
  • SpinGroup#wait loops until every task's future completes — so it spins forever.

This sounds exotic until you look at what lives outside StandardError: NotImplementedError and LoadError are both ScriptError subclasses. A failed require inside any spinner task froze the whole CLI with an eternal spinner (verified against main).

WorkQueue fixes

  • Workers rescue Exception and fail the future, so waiters always observe the error; the worker stays alive for subsequent tasks. Interrupt handling is unchanged (fail + re-raise to terminate the worker).
  • Every future is settled structurally, from an ensure. Thread#raise and Thread#kill can land outside the worker's rescues (e.g. WorkQueue#interrupt hitting a worker between Queue#pop and the begin). A future abandoned that way is failed with WorkQueue::AbandonedTaskError instead of blocking Future#value forever.
  • interrupt tolerates the Interrupt that Thread#join can re-raise. The Interrupt it raises into a worker can land after that worker has already left its rescues (it was terminating after the queue closed); the worker dies with an unhandled Interrupt, and join would re-raise it in the interrupting thread — replacing whatever that thread was propagating (e.g. the SystemExit that triggered the interrupt).
  • WorkQueue.new rejects max_concurrent < 1, which produced a queue that accepts work, never starts it, and returns from #wait immediately while futures stay incomplete — a deadlock for any Future#value caller.

SpinGroup fixes

  • Task#check treats any exception other than Interrupt/SystemExit as a task failure, reported through the normal debrief. (An earlier revision raised non-StandardErrors out of SpinGroup#wait; review pointed out that skips work_queue.interrupt, the final render, and the debrief, leaving sibling workers running after wait returns.)
  • wait interrupts the queue before letting SystemExit propagate, so a stray exit in a task doesn't orphan the remaining workers.
  • initialize falls back to 1024 for any non-positive max_concurrent instead of raising from WorkQueue's new guard, consistent with the documented 0 = unlimited.

Progress fixes

  • tick validates percent/set_percent explicitly — finite numbers only, with the offending argument named in the ArgumentError — instead of relying on Array#min raising on an incomparable value. Validation happens before assignment, so a raising tick leaves the bar renderable from its last valid state (previously the next render raised a confusing secondary error far from the actual mistake).
  • The result is clamped to 0.0..1.0. The lower bound was previously unguarded, so a negative percent rendered a negative suffix.

Tests

  • Worker survives a ScriptError: future completes, value re-raises, a subsequent task on the same worker still runs.
  • A worker that exits without finishing (Thread.exit) still settles its future, with AbandonedTaskError.
  • SpinGroup: a NotImplementedError task is reported through failure_debrief with its sibling still completing; wait returns false promptly instead of hanging (with a Timeout guard so a regression fails the suite rather than wedging CI).
  • SpinGroup: SystemExit propagates out of wait with its exit status intact, and remaining workers are interrupted rather than orphaned.
  • SpinGroup runs tasks for max_concurrent: 0 and -1.
  • ArgumentError for WorkQueue.new(0) / (-1).
  • Invalid tick values (String, NaN, ±Infinity, Array) raise without corrupting the bar's last valid state; error messages name the bad argument; negative values clamp to 0%.

🤖 Generated with Claude Code

(posted by an LLM bot on behalf of Gord)

@GoodForOneFare
GoodForOneFare force-pushed the gordo-work-queue-progress-guards branch from 4bf35b7 to 837a51b Compare August 14, 2026 19:38
@GoodForOneFare GoodForOneFare changed the title WorkQueue: complete futures on any exception; validate WorkQueue/Progress input Settle futures on any exception; validate WorkQueue/Progress input Aug 14, 2026
Settle futures exactly once while deferring asynchronous interrupts during state changes. Track active workers separately from joinable thread history, queue work before checking capacity, and replace workers that die while work remains.

Let fatal task exceptions terminate their worker after failing the future, clamp non-positive WorkQueue concurrency to one, and preserve SpinGroup’s documented zero-as-default behavior. SystemExit now interrupts only queues owned by the SpinGroup.

Remove the unrelated Progress validation changes and add regressions for Thread.exit recovery, worker-initiated interruption, shared queues, and observable SystemExit cleanup.

Co-authored-by: River <river@shopify.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Assisted-By: devx/67971778-53ac-43ed-9659-234fe50f4cd9
@GoodForOneFare
GoodForOneFare force-pushed the gordo-work-queue-progress-guards branch from 837a51b to 1b9a760 Compare August 14, 2026 20:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant