refactor(signaling): explicit signal lifecycle state machine - #1402
refactor(signaling): explicit signal lifecycle state machine#1402lukasIO wants to merge 11 commits into
Conversation
SignalInner tracked its lifecycle in two fields, a stream slot and a `reconnecting` AtomicBool, that had to be kept in step by hand across restart, set_reconnected, close and send. It now holds one SignalState that owns the transport: Connected, Reconnecting, Offline, Disconnecting, Closed. Every change goes through SignalState::transition, one match that is the whole table and also says which transport each move releases. An input a state does not accept is logged and refused; a resume from a state that cannot accept it fails with SignalError::InvalidState. The resume gate stays a state until the engine confirms, a stale transport cannot report in because SignalClient::restart awaits the old task first, and ReconnectFailed always lands in Offline. The held-signal queue moves to a sync parking_lot Mutex that is never held across an await. The old async queue lock was taken in both orders relative to the stream lock, which with tokio's fair RwLock could deadlock against a pending restart writer. A send that fails with any transport error is now held like a SendError was. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Changeset ✓This PR includes a changeset covering all affected packages:
|
…it/rust-sdks into lukas/signal-state-machine
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Note
This report is out of date. Scroll down for Devin Review's latest report on this PR.
Devin Review found 3 potential issues.
2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
| let mut state = self.state.write().await; | ||
| let old_stream = state.transition(SignalInput::Reconnect).map_err(|_| { | ||
| SignalError::InvalidState(format!("resume the signal session from {state:?}")) | ||
| })?; |
There was a problem hiding this comment.
🟡 Cancelled reconnects block every retry
If restart is cancelled after Reconnect, the state remains Reconnecting(None). Every later retry returns InvalidState, leaving the client unable to reconnect.
Learn more
Make SignalClient::restart and SignalInner::restart cancellation-safe. Once the state transitions to Reconnecting(None), dropping the future must restore a retryable state and clean up any old or newly opened transport. The close phase has the same issue after entering Disconnecting. Consider an RAII transition guard or redesigning the lifecycle operation as an owned task whose cleanup completes independently of the caller future. Add tests that poll restart into the close and connect phases, cancel it, and verify a subsequent restart can proceed.
Was this helpful? React with 👍 or 👎 to provide feedback.
1egoman
left a comment
1egoman
left a comment
There was a problem hiding this comment.
Generally makes sense to me. I focused more on the infrastructure and general approach, and less on the exact states and all the exact transitions between them.
One thing missing which I was expecting to see: some sort of mechanism that a visualizer could subscribe to state changes and use them to build a visualization. Was this expected to be part of a follow up change or did I miss it in here somewhere?
| pub async fn close(&self, notify_close: bool) { | ||
| if let Some(stream) = self.stream.write().await.take() { | ||
| // Already closing or closed: whoever owns that close finishes it. | ||
| let Ok(stream) = self.state.write().await.transition(SignalInput::Close) else { | ||
| return; | ||
| }; | ||
| if let Some(stream) = stream { | ||
| stream.close(notify_close).await; | ||
| } | ||
| let _ = self.state.write().await.transition(SignalInput::CloseComplete); | ||
| } | ||
|
|
There was a problem hiding this comment.
thought: I'm not sure about this idea or not, but I wonder if there would be benefit to driving the state machine side effects by subscribing to a stream of state changes from the state machine, not just running the side effects alongside at every .transition(...) call sizte. What this would look like is some handler consuming the stream of state changes and that handler containing a big match not unlike how .transition(...) works today.
I will say one nice thing about that is it means that places which all issue the same .transition(...) call would be guaranteed to run the same side effects, which could be nice.
There was a problem hiding this comment.
Yeah, agree, I would like to move into that direction eventually.
I think it should be rather easy to transition (pun intended) to that design later on once we also represent the PC connection as more of a state machine pattern?
There was a problem hiding this comment.
Makes sense, cool, let's revisit it later then.
yeah, that's intended as follow up work |
The machine now starts in Connecting, which is also its Default, so the transition table can use mem::take instead of a placeholder. The constructor drives the first transition before there is a client to hold the state: ConnectComplete(stream) lands in Connected, and any failure, transport or missing JoinResponse, lands in Closed and is dropped with the error. The v1/v0 transport selection moves out of connect into open_transport so the constructor reads as: open, join, transition, build. Behaviour is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…it/rust-sdks into lukas/signal-state-machine
There was a problem hiding this comment.
Devin Review found 1 new potential issue.
2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
| let queued = std::mem::take(&mut *self.queue.lock()); | ||
| for signal in queued { | ||
| if let Err(err) = stream.send(signal).await { | ||
| log::error!("failed to send queued signal: {}", err); // Lost message | ||
| } |
There was a problem hiding this comment.
🟡 Queued signals can be overtaken
While flush_queue awaits each held signal, concurrent send calls can transmit newer requests between them. Deferred session mutations reach the server out of order.
Learn more
flush_queue removes the entire queue before performing any asynchronous sends. The state read lock does not serialize senders, because multiple readers can hold it concurrently. After the first held signal awaits its acknowledgement, another send sees an empty queue and can enqueue its newer signal before the next held signal. This contradicts the ordering required by the connected send path.
Example: The queue contains Mute(track, true) followed by Mute(track, false). Flushing sends the first mutation, then a concurrent newer Mute(track, true) can enter before the second. The server ends with the older unmute state instead of the latest mute state.
Recommended fix: Serialize queue draining and direct sends with an async send gate, or move sending into one actor-owned FIFO. Keep the synchronous queue lock out of awaits while preventing connected senders from entering until the drained batch finishes.
Was this helpful? React with 👍 or 👎 to provide feedback.
2 participants
SignalInner tracked its lifecycle in two fields, a stream slot and a
reconnectingAtomicBool, that had to be kept in step by hand across restart, set_reconnected, close and send. It now holds one SignalState that owns the transport: Connected, Reconnecting, Offline, Disconnecting, Closed.Every change goes through SignalState::transition, one match that is the whole table and also says which transport each move releases. An input a state does not accept is logged and refused; a resume from a state that cannot accept it fails with SignalError::InvalidState. The resume gate stays a state until the engine confirms, a stale transport cannot report in because SignalClient::restart awaits the old task first, and ReconnectFailed always lands in Offline.
The held-signal queue moves to a sync parking_lot Mutex that is never held across an await. The old async queue lock was taken in both orders relative to the stream lock, which with tokio's fair RwLock could deadlock against a pending restart writer. A send that fails with any transport error is now held like a SendError was.