starknet_transaction_prover: track saturation and queue depth on the request path

[avi-starkware]

Adds the state /health will read in the next PR: a SaturationMonitor that
opens a window on the first busy-reject and closes it on any forward progress --
a request acquiring a worker slot, or a slot being released. Slot release has to
count, because once 503 pulls the load balancer's traffic away, a clear that
waited for a new request would latch 503 forever; a drop guard covers the
success, error and client-disconnect exits alike.

Both busy-reject arms go through record_busy_reject, which counts the outcome,
opens the window, warns and returns -32005 together, so a reject cannot count
without latching. Also exposes queue depth and queue-wait duration, bucketed at
the wait timeout.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com

[reviewable-StarkWare]

This change is 

[avi-starkware]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• starknet_transaction_prover: log resolved config at startup (secrets redacted) #14684
• starknet_transaction_prover: origin-level breadcrumbs for validation errors #14172
• starknet_transaction_prover: /health returns 503 when service is saturated #14171
• starknet_transaction_prover: track saturation and queue depth on the request path #15027 👈 (View in Graphite)
• starknet_transaction_prover: bump prover_panics_total on panic hook fire #14170
• starknet_transaction_prover: HTTP request count + latency + in-flight metrics #14169
• starknet_transaction_prover: proving-job duration + outcome metrics #14168
• starknet_transaction_prover: Prometheus /metrics endpoint with build_info #14167
• starknet_transaction_prover: extract the TLS accept loop and test its drain #15049
• starknet_transaction_prover: graceful shutdown on SIGTERM/SIGINT #15026
• starknet_transaction_prover: global panic hook with structured log #14166
• starknet_transaction_prover: document existing logging, request ids and health probe in README #14203
• main-v0.14.4

This stack of pull requests is managed by Graphite. Learn more about stacking.

[cursor]

PR Summary

Medium Risk
Changes admission/queue behavior instrumentation and shared RPC construction (from_config now takes SaturationMonitor); reject paths and drop ordering affect overload signaling, though successful proving paths are unchanged aside from metrics and guards.

Overview
Adds a SaturationMonitor on the proving request path so a follow-up can drive /health from sustained overload: the window opens on the first busy-reject (-32005) and clears on forward progress (worker slot acquired or released via RAII SaturationClearGuard), with careful drop ordering so slot release cannot leave a stale saturation latch.

Busy-rejects (queue full vs queue wait timeout) now share record_busy_reject, which increments prover_prove_transaction_outcome_total with rejected_queue_full / rejected_wait_timeout, opens the saturation window, and logs—so counting and saturation stay aligned.

Observability: prover_queue_waiting_requests and prover_queue_wait_duration_seconds (with tuned histogram buckets) track FIFO wait behind worker slots; README and exporter pre-registration are updated accordingly. Tests cover reject paths, gauge behavior, and saturation window semantics without a full prove run.

^{Reviewed by Cursor Bugbot for commit f6e04fe. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1d3075e. Configure here.}

[cursor]

Saturation latch on admission drop order

Medium Severity

_admission is declared before the clear guard, so it drops after mark_progress. A concurrent queue-full reject can reopen the saturation window in that gap with no SaturationClearGuard left to clear it. With max_queued_requests = 0 and max_concurrent_requests = 1, admission stays full until that late drop, so the window can latch open once load-balancer traffic drains — the failure mode this binding order was meant to prevent.

 

^{Reviewed by Cursor Bugbot for commit 1d3075e. Configure here.}