feat: external project agent dispatch + Twilio SMS operator routing - #5885
Draft
mfethe1 wants to merge 26 commits into
Draft
feat: external project agent dispatch + Twilio SMS operator routing#5885mfethe1 wants to merge 26 commits into
mfethe1 wants to merge 26 commits into
Conversation
added 26 commits
August 14, 2026 11:56
…erator agent Research-backed plan for routing Buzz messages/SMS into agent dispatch against bidcraft (BuildBid) and construct-pro, plus the Twilio inbound/ outbound SMS bridge and operator-agent disambiguation flow. Signed-off-by: Michael Feth <michael@jira-flow.com>
Add --project-paths (project id -> local checkout path) and --channel-projects (channel UUID -> project id) config, resolved once at startup into PromptContext.channel_cwd. Dispatched sessions in a bound channel now launch with that project's cwd instead of always the harness's own working directory; unbound channels are unaffected. Slices 1+2 of docs/wayfinder/map-external-project-sms-integration.md, combined because an unconsumed resolver trips this crate's deny(dead_code) clippy gate. Signed-off-by: Michael Feth <michael@jira-flow.com>
…/paths - buzz-cli already supports channel<->project binding (projects update --channel), verified by reading the persist/read path and its existing test - no new CLI subcommand needed, correcting slice 3's scope. - E:/Projects/buildbid has no top-level .git; the real bidcraft checkout is E:/Projects/buildbid/bidcraft-repo (verified: origin points at mfethe1/bidcraft, real commit history). - Flag slices 4/5/11 (live agent dispatch) as needing a supervised session rather than unattended background execution - they spin up a real relay + harness + agent subprocess and, for construct-pro, write into a shared repo other agents are actively working in. Signed-off-by: Michael Feth <michael@jira-flow.com>
Schema for the Twilio SMS bridge (slice 6 of the external-integration design doc): phone_number is the allow-list key, default_project names the NIP-MP project d-tag the sms-operator persona should dispatch into when unambiguous. E.164 and 32-byte-pubkey CHECK constraints catch malformed inserts at the DB layer rather than trusting callers. Verified: crate builds clean (sqlx::migrate! embeds the file without macro errors). Not yet applied against a live Postgres in this pass — enforcement logic (the actual allow-list check) lands with the inbound webhook handler next, since that's where it's consumed. Signed-off-by: Michael Feth <michael@jira-flow.com>
…llow-list Adds POST /hooks/sms/inbound: validates X-Twilio-Signature (HMAC-SHA1 per Twilio's documented scheme, crates/buzz-relay/src/twilio_auth.rs), then requires the sender's phone number be allowed=true in the sms_identities table (crates/buzz-db/src/sms.rs) before doing anything else. Both failures return the same generic 403 so the endpoint can't be used as an oracle for probing valid signatures or registered numbers. twilio_auth's test vectors were computed independently outside this codebase (openssl dgst -hmac + Python's hmac/hashlib, both agreeing) rather than hand-derived from the implementation, so the tests verify the algorithm, not just internal self-consistency. Stops at "allowed -> 200 OK acknowledged" -- does not yet synthesize a relay event (KIND_STREAM_MESSAGE_V2) from the message, which is the next slice. Not yet verified against a live Postgres or a real Twilio account (TWILIO_AUTH_TOKEN/TWILIO_WEBHOOK_URL unset fails closed). Also updates docs/wayfinder/map-external-project-sms-integration.md with a Privacy section: NIP-MP project visibility defaults to `listed` (buzz-cli/src/lib.rs:1262-1263) and channel visibility defaults to `open` (channel_templates.rs:60) -- both must be explicitly overridden to `unlisted`/`private` for bidcraft/construct-pro given they're private repos. Also flags the fetch-before-dispatch freshness gap as a new tracked task rather than silently leaving it undocumented. Signed-off-by: Michael Feth <michael@jira-flow.com>
Replaces the prior "allowed -> 200 OK, stop" placeholder in crates/buzz-relay/src/api/sms.rs with real event synthesis: builds a KIND_STREAM_MESSAGE (kind 9) event tagged h (SMS-inbox channel), sms_from, sms_sid, p (sender's linked pubkey when known), and project (the sender's default_project when set), signs it with the relay's own keypair, and inserts it via Db::insert_event_with_thread_metadata. Uses KIND_STREAM_MESSAGE (9), not KIND_STREAM_MESSAGE_V2 (40002) as the design doc originally guessed -- corrected after checking buzz-acp's default Mentions-mode subscribe filter (resolve_channel_filters in buzz-acp/src/config.rs), which only wakes agents on kind 9. A V2-only event would never trigger the sms-operator persona meant to read it. Mirrors workflow_sink.rs's proven pattern for a relay-authored message (same signing/tagging/insert approach), rather than inventing a new one. New crates/buzz-relay/src/config.rs field: twilio_sms_inbox_channel (env TWILIO_SMS_INBOX_CHANNEL) -- a single global inbox channel for v1; multi-community routing is a documented future enhancement. Verified: build_tags() is a pure, unit-tested function (6 tests covering h/sms_from/sms_sid presence, p/project inclusion/omission, and malformed-pubkey handling) -- cargo clippy -D warnings and cargo test both confirmed clean with explicitly-checked exit codes, not piped. NOT verified: the actual DB insert path (insert_event_with_thread_metadata) against a live Postgres -- no DB infra was available this session. Signed-off-by: Michael Feth <michael@jira-flow.com>
Slice 7's event synthesis is real and unit-tested now (see c3a70ce). Re-scopes task 8 precisely: the sms-operator persona is a real .persona.md pack (not .toml), and buzz-acp needs new per-message project-tag routing since the SMS-inbox is one shared channel across all senders -- today's --channel-projects binding is per-channel only, not per-message. Signed-off-by: Michael Feth <michael@jira-flow.com>
…ce-8 plan Both found by an adversarial research pass, then independently re-verified against real code before recording. 1. Persona Packs are inert at runtime. buzz-acp never loads one: zero buzz_persona references in buzz-acp/src despite a declared Cargo dependency, no --pack/--persona flag, and a persona's subscribe: field is parsed but never mapped to subscribe config. Writing sms-operator.persona.md would validate cleanly and do nothing. Deploying an agent today means desktop personas.json or setting buzz-acp env vars directly. 2. Per-message project routing is blocked by session caching. Reading the project tag IS small (batch is in scope at pool.rs:1684), but pool.rs:1678 short-circuits to a cached session that never reaches the resolver, and cwd is immutable after session/new. A naive implementation routes the first message correctly then silently ignores the tag forever -- and passes a single-message test, the worst failure shape. Neither implemented, deliberately. Landing either half as originally scoped would look green and be wrong. Signed-off-by: Michael Feth <michael@jira-flow.com>
Two blockers recorded in db2747d are now fixed. 1. Persona packs were inert -- buzz-acp declared buzz-persona as a Cargo dependency and never called it, so a pack validated cleanly and changed nothing. Adds --pack/--persona resolution at startup, mapping a ResolvedPersona onto Config (system prompt, model, title, runtime, instructions, env, subscribe). Explicit CLI/env values win over pack values via clap ArgMatches value_source, so a pack only supplies defaults. Implements the '#'-stripping for subscribe channel names that PERSONA_PACK_SPEC promised and no code did. A bad pack path is a hard startup error, never a silent no-op. 2. Per-message project routing was defeated by session caching: the cached-session short-circuit never reached the cwd resolver and a session's cwd is immutable after session/new, so a naive tag-read would route message 1 and silently ignore the tag forever -- while passing a single-message test. Adds resolve_turn_routing() + batch_project_tag() and session_cwds on SessionState, invalidating pre-emptively only when the resolved cwd actually changes, ordered before the core-memory and canvas/title blocks so a replacement session rebuilds them. Security: a project id now arrives from an event tag, i.e. attacker-influencable. It is only ever a lookup key into the operator-configured --project-paths map and is never joined into a path; unknown ids fall back to the channel binding. Verified by re-running the gates directly rather than trusting the build agents: 29 config::persona_pack_tests pass, 6 pool::tests:: routing_* pass (including routing_hostile_project_ids_cannot_escape_ the_configured_map, which drives 17 hostile ids -- traversal, absolute, UNC, null byte, $HOME/%USERPROFILE% -- and also asserts a legitimate id still resolves so it is not vacuously rejecting everything), and second_message_with_a_different_project_tag_creates_ a_session_in_the_new_cwd passes -- a three-turn test capturing real ACP wire traffic that would fail if routing regressed to first-message-only. cargo fmt --check and cargo clippy -D warnings both clean. Not verified: no live relay, harness, or agent process was run. Signed-off-by: Michael Feth <michael@jira-flow.com>
The "packs are inert" and "session caching defeats per-message routing" findings were true when written and are now fixed. Records which tests cover what, that verification is unit-level only (no live relay/harness/agent has been run), the new attacker-influencable project-tag surface and how it is contained, and the C:\-full linker workaround needed to build on this machine. Signed-off-by: Michael Feth <michael@jira-flow.com>
st.txt and 9 malformed Windows-temp-path files had accumulated as untracked cruft from a pool.rs test bug (fixed separately) that wrote capture files to literal, unresolved path strings. Add a safety-net ignore rule in addition to the code fix so any recurrence doesn't show up as noise in git status again. Signed-off-by: Michael Feth <michael@jira-flow.com>
…Windows Four tests (standing-context, channel-delivery, merged-cancel-prompt, late-steer-ack lifecycle tests) spawned "bash" for their fake-ACP-agent shell script, which on this machine resolves to WSL bash -- WSL mounts drives at /mnt/e/, not E:/, so a Windows temp path like E:\Users\...\Temp\foo.ndjson is neither a valid WSL path nor a POSIX path it can create. The path's backslashes and colon got baked into a single literal filename instead, producing the malformed on-disk artifacts cleaned up in the previous commit, and the capture file at the real path was never written -- causing the same 4 tests to fail with NotFound when reading it back. Switch these tests from "bash" to "sh", matching the already-working pattern in spawn_fake_session_agent (which has a comment explaining exactly this WSL-vs-toolchain-sh distinction), and normalize backslashes to forward slashes in the quoted capture path before shell-escaping it, matching that same helper. The scripts are already portable POSIX with no bashisms, so no other changes are needed. Verified: all 4 tests now pass (previously 0/4); full buzz-acp suite went from 810 passed/24 failed to 823 passed/11 failed, with the remaining 11 failures confined to unrelated pre-existing acp.rs steer/timing races. cargo fmt --check and clippy -D warnings both clean. Signed-off-by: Michael Feth <michael@jira-flow.com>
Prior test-status claims in this doc were accurate for the specific tests they named but implied the crate was fully green when it wasn't: cargo test -p buzz-acp was 810 passed / 24 failed, including 4 failures that were a real bug in this crate's own tests (fixed in 18bd342). Records the corrected 823/11 count and the lesson to state which tests pass rather than imply the whole suite does. Signed-off-by: Michael Feth <michael@jira-flow.com>
Per-message project-tag routing already existed in buzz-acp (resolve_turn_routing/batch_project_tag) and the [Buzz event] block already renders an event's raw tags verbatim (format_event_block in queue.rs), so the remaining gap for slice 8 was purely the persona itself: a system prompt that reads the project tag and decides fast path vs disambiguation, per docs/wayfinder/map-external-project-sms-integration.md Part B step 5. Fast path: project tag present and recognized (bidcraft | construct-pro) -- the harness has already placed the agent in that project's cwd, so it just does the requested work and replies with --reply-to so the outbound sms_sink (not yet built) can find it. Ambiguous path: tag absent or unrecognized -- do no work, ask the sender to pick 1/2, and explicitly do not pretend a follow-up reply can re-route this session: routing already happened server-side before dispatch, and there is no mechanism today for the agent to change which cwd it was placed in or to write back to the sms_identities.default_project column itself. That gap is real and called out in the prompt rather than glossed over. Verified: `buzz pack validate`/`buzz pack inspect` on the new pack both clean, and (beyond unit-level checks) a real `buzz-acp --pack crates/buzz-persona/packs/sms-operator` invocation logs "loaded persona pack pack=com.buzz.sms-operator persona=sms-operator" before failing later on an unrelated missing `goose` binary on this machine. Signed-off-by: Michael Feth <michael@jira-flow.com>
Records the sms-operator persona pack (60d59f9) closing slice 8, and narrows slice 9 to its real remaining gap: nothing today lets a sender's reply update sms_identities.default_project, so disambiguation is currently one-directional. Signed-off-by: Michael Feth <michael@jira-flow.com>
… 12)
--project-paths/--channel-projects point at a local checkout the
operator already has, but nothing previously fetched or checked its
freshness before dispatching an agent into it -- a checkout that fell
behind its remote (e.g. someone else pushed) would be worked on
silently as if it were current.
Adds commits_behind_upstream(), run once per new session creation
(not per turn -- a network fetch on every message would be far too
slow), right where effective_cwd is already known in the channel
session-creation branch. Only runs `git fetch` (updates
remote-tracking refs) and `git rev-list --count HEAD..@{u}` --
deliberately never fast-forwards, merges, or pulls on the operator's
behalf, since the checkout may have uncommitted work or be in use by
another agent concurrently. Every failure path (no .git, no remote,
offline, no upstream branch, git missing) is fail-open and silent,
since we can only warn about a comparison we could actually make.
Gated on effective_cwd != ctx.cwd so the harness's own default cwd
never pays this cost.
Verified with 4 new tests against real temp git repos (init, clone,
push, fetch) rather than mocks: not-a-repo and no-remote both resolve
to "can't tell" (None), a fresh clone with no new upstream commits
reports Some(0), and two commits pushed after cloning are correctly
counted as Some(2). Full buzz-acp suite: 827 passed / 11 failed (same
11 pre-existing unrelated acp.rs races as before this change -- no
regressions). cargo fmt --check and clippy -D warnings both clean.
Signed-off-by: Michael Feth <michael@jira-flow.com>
Records the fetch-before-dispatch freshness check (563b85a). Signed-off-by: Michael Feth <michael@jira-flow.com>
Closes the SMS loop's return path: a reply posted into the SMS-inbox channel that threads back to an inbound SMS is now sent to the original sender as a text. Resolution is deliberately narrow. maybe_send_outbound_sms fires only when the event is in the configured inbox channel, does NOT itself carry sms_from (an inbound event must never be echoed back to Twilio as if it were a reply), and has an e tag resolving to a stored event that does carry sms_from -- that tag is where the destination number comes from, so a reply can only ever text the number that started the thread. Resolution is one hop only and does not walk a thread; a deeper reply simply won't send, since widening it would let any reply anywhere in an SMS-rooted thread trigger an outbound text. Hooked into dispatch_persistent_event_inner, the same post-persist seam workflow triggering uses. Spawned rather than awaited so a slow or unreachable Twilio can't stall event dispatch, and fail-open throughout -- every outcome is a SendOutcome variant, never an error the caller must handle. Adds TWILIO_ACCOUNT_SID / TWILIO_FROM_NUMBER (both required to send; absent means skip with a warning rather than fail) and TWILIO_API_BASE_URL, which exists so tests can point sends at a local mock instead of the real API. The auth token is the one already used for inbound signature validation. The form body is built via url::form_urlencoded rather than reqwest's .form(): the workspace pins reqwest with default-features off, so the urlencoded feature isn't compiled in. Verified with 8 new tests, 3 of which drive a real HTTP round-trip against a local single-shot mock listener and assert the actual wire format -- POST path carrying the account SID, basic auth header, form-urlencoded content type, and percent-encoded To/From/Body -- plus non-2xx and unreachable-host paths both landing on Failed. That caught a real detail on the way in (hyper lowercases header names on the wire). Full buzz-relay suite: 883 passed / 11 failed, and those same 11 fail identically on a stashed clean baseline (admin, media, git-policy, telemetry -- all unrelated), so no regressions. Not verified: no send against real Twilio credentials, and no live relay run. Signed-off-by: Michael Feth <michael@jira-flow.com>
Records the outbound sms_sink (04bf428), and that its original live-Twilio verification is still outstanding -- that needs a provisioned account/number and rolls into slice 11. Signed-off-by: Michael Feth <michael@jira-flow.com>
Manual conflict resolution in pool.rs during the rebase onto main left one line over the fmt width limit. No behavior change. Signed-off-by: Michael Feth <michael@jira-flow.com>
…sion Rebasing onto main pulled in 0031_workflow_run_error_codes.sql (from "Make workflow run history authoritative in Desktop", block#5780), which collided with this branch's 0031_sms_identities.sql -- two migrations sharing one version number. Notably this still COMPILED clean: sqlx::migrate! builds the migrator without complaint, so the collision would only have surfaced against a live database at relay startup, i.e. at deploy time. Renumbering to 0032 is safe with no migration-history consequences because sms_identities has never been applied to any live Postgres. Signed-off-by: Michael Feth <michael@jira-flow.com>
The SMS bridge added six config variables that .env.example never mentioned, so the only way to discover them was reading config.rs. Documents each one alongside its failure mode: unset auth token means inbound rejects every request rather than skipping validation, unset inbox channel means inbound is rejected rather than silently dropped, and unset account SID / from number disables outbound sending. Also records why TWILIO_WEBHOOK_URL is configured rather than derived from request headers -- Twilio signs the literal configured string, and behind a proxy the headers can disagree with it. Signed-off-by: Michael Feth <michael@jira-flow.com>
…t prove Deployed to isolated Railway infra and verified against a running relay: migration 0032 applied to a real Postgres, the inbound route exists (403/415 rather than the upstream image's 404), and unsigned vs bogus-signature requests return byte-identical 403s -- the non-oracle property, diffed rather than eyeballed. States the precise limit of that evidence rather than overclaiming: the 403s currently come from the "Twilio not configured" guard, so this verifies fail-closed-when-unconfigured, NOT that a genuine Twilio signature is accepted. That still needs real credentials. Also records two deployment traps: Railway's redeploy re-runs the same snapshot instead of rebuilding the configured branch (it silently redeployed stale main with every layer cached, indistinguishable from a real build), and the git conformance probe is a hard startup gate needing S3 that must be disabled on SMS-only relays. Signed-off-by: Michael Feth <michael@jira-flow.com>
Real Twilio credentials against the deployed test relay. Records the three things now actually proven rather than assumed: a genuine HMAC-SHA1 signature is accepted (only the reject path had been exercised before), event synthesis produces a correct relay-signed kind:9 with h/sms_from/sms_sid tags, and -- the decisive one -- the allow-list genuinely gates, shown by the same signed request returning 403 before the sms_identities row existed and 200 after, with the DB row as the only changed variable. That single pair rules out both "it accepts anything signed" and "the 403 was really a signature failure". Also notes the absent project tag is correct (default_project NULL = the ambiguous case), and that seeding the row needed railway ssh because nothing writes sms_identities -- the known slice-9 gap. Outbound remains unverified and is likely blocked outside our code by a FAILED A2P campaign (30909), which would surface as carrier filtering (30034) rather than an sms_sink defect. Signed-off-by: Michael Feth <michael@jira-flow.com>
The inbound SMS handler persisted its synthesized event and returned, never calling dispatch_persistent_event. The event therefore existed in the database -- every read path, including `buzz messages get`, showed it -- but was never fanned out to live WebSocket subscribers, never published to Redis, and never triggered workflows. Consequence: an ACP agent subscribed to the SMS inbox never woke, so the entire operator-agent flow silently did nothing while every individual piece looked correct. Found by running it end to end: the webhook returned 200, the event was readable in the channel, and the harness sat idle with no error anywhere. workflow_sink.rs -- the pattern this handler was written to mirror -- makes exactly this call after its own insert. This adds it, gated on was_inserted for idempotency, resolving the TenantContext from the identity's community the same way workflow_sink does. Delivery failures are logged rather than surfaced: the message is already durably accepted at that point, matching how the WebSocket and REST ingest paths treat post-commit delivery. Verified: cargo fmt --check and clippy -D warnings clean; buzz-relay 887 passed / 10 failed, all pre-existing and unrelated (admin, media, git-policy, telemetry) on the same baseline as before this change. Signed-off-by: Michael Feth <michael@jira-flow.com>
Inbound -> signature -> allow-list -> event synthesis -> fan-out -> agent dispatch -> persona decision -> threaded reply -> sms_sink -> Twilio API accept. Every stage this codebase owns is now exercised live; only carrier delivery fails (error 30034, the FAILED A2P campaign), which is outside the code. Notable: the persona took the correct ambiguous branch. With the sender's default_project NULL the event carried no project tag, and the agent posted the specified disambiguation prompt rather than guessing a project. The reply threaded back with an e-tag resolving to the originating sms_sid -- exactly what sms_sink keys on -- and sms_sink then logged a successful send. Records two traps that cost real time. First, an earlier run looked like the fan-out fix had failed (200 returned, agent idle) but was a race in the test: the webhook landed 7s before the harness finished subscribing, and live fan-out only reaches connected subscribers. Diagnosing it required the relay HTTP access log to compare event-arrived-at against subscriber-connected-at. Second, claude-code-acp refuses to start with CLAUDECODE inherited, failing as a -32603 that reads like an ACP protocol fault rather than an environment guard. Signed-off-by: Michael Feth <michael@jira-flow.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds two related capabilities, designed together because the second depends on the first:
Design doc:
docs/wayfinder/map-external-project-sms-integration.md(included in this branch).Approach
Reuses existing primitives rather than inventing new ones. Project scoping uses the existing NIP-MP
kind:30621project primitive and the existingbuzz projects update --channelbinding — no new event kind, no new CLI subcommand. Inbound SMS synthesizes a plainkind:9KIND_STREAM_MESSAGE(deliberately not V2: buzz-acp's default Mentions-mode subscribe filter only wakes agents on kind 9, so a V2-only event would never trigger the persona meant to read it).Routing is per-message, not per-channel. The SMS inbox is one shared channel across all senders and projects, so a static channel→project binding isn't sufficient.
resolve_turn_routing()reads a per-messageprojecttag; because a session'scwdis immutable aftersession/new, it also trackssession_cwdsand pre-emptively invalidates a session only when the resolved cwd actually changes — ordered before the core-memory and canvas/title blocks so a replacement session rebuilds them.Security notes
--project-pathsmap and is never joined into a filesystem path; unknown ids fall back to the channel binding.routing_hostile_project_ids_cannot_escape_the_configured_mapdrives 17 hostile ids (traversal, absolute, UNC, null byte,$HOME,%USERPROFILE%, case/whitespace variants) and also asserts a legitimate id still resolves, so it isn't vacuously rejecting everything.sms_fromtag on the event being replied to. An inbound event is explicitly never echoed back to Twilio as if it were a reply.Testing
Unit-level throughout, including tests that drive real behavior rather than mocks:
commits_behind_upstreamis tested against real temporary git repos (init/clone/push/fetch): not-a-repo and no-remote resolve to "can't tell", a fresh clone reports 0 behind, two commits pushed after cloning are correctly counted as 2.To/From/Body), plus non-2xx and unreachable-host paths.second_message_with_a_different_project_tag_creates_a_session_in_the_new_cwdis a three-turn test capturing real ACP wire traffic, asserting exactly twosession/newcalls with the second in the new cwd. It fails if routing regresses to first-message-only — which was the specific trap here, since a naive implementation passes a single-message test while silently ignoring the tag forever after.cargo fmt --checkandcargo clippy -D warningsare clean. Full-suite counts:buzz-acp829 passed / 11 failed,buzz-relay886 passed / 11 failed — those failures are pre-existing and unrelated (ACP steer/timing races; admin/media/git-policy/telemetry), confirmed by running the same suites against a stashed clean baseline.Not verified
Stated plainly because it matters for review:
sms_identitiesDB insert path has not executed against a live Postgres.Known gap
Disambiguation is currently one-directional. The operator persona can ask "which project?", but nothing in the codebase lets a sender's reply update
sms_identities.default_project— that column is operator-set. The persona is written to be honest about this rather than pretending a "reply 1" re-routes it. Closing the loop needs a new CLI/admin surface and is out of scope here.