Repository navigation
feat(mcpbroker): authorize gateway calls with the identity broker and send them with a per-call token - #2818
ChrisJBurns wants to merge 18 commits into
Conversation
…nts outside I-JSON
argumentsSHA256 canonicalizes params.arguments per RFC 8785 using only the
standard library and refuses what the vMCP could hash differently: invalid
UTF-8, lone surrogate escapes, duplicate member names, non-object values,
non-finite numbers, and integers beyond 2^53. Absent or null arguments hash
and are sent as {}.
Co-authored-by: Mecatl <noreply@mecatl.dev>
…MCP broker Co-authored-by: Mecatl <noreply@mecatl.dev>
…n, and dropped connections Co-authored-by: Mecatl <noreply@mecatl.dev>
Co-authored-by: Mecatl <noreply@mecatl.dev>
… send them with a per-call token A tools/call on a broker route whose server is a gateway server is authorized by the identity broker for the acting agent and sent to the gateway with a gateway token for that call alone. The argument hash is checked first; one svid_expired renews the agent credential and asks again; one invalid_token is retried reporting the rejected attempt; a call is claimed in a send-once record before it is sent and released only when no attempt may have run. Refusals and unknown outcomes are model-visible tool errors with a code. A request the gateway sends during the call is declined and the call's real answer is returned with a note. Gateway routes no longer ask for the MCP broker's OAuth authorization at call time. Co-authored-by: Mecatl <noreply@mecatl.dev>
NewStreamingClient serves requests from an http.Handler in process and streams the response. As net/http's server does, the writer buffers: the response is sent, and RoundTrip returns, at the handler's first Flush, once 4 KiB is written, or when the handler returns, and each later Flush sends what was written since. The caller's context, or closing the body, cancels the handler; a body read after the context ended fails with its error. A handler panic drops the response. Co-authored-by: Mecatl <noreply@mecatl.dev>
…way is configured When the identity broker has a gateway entry and the bundled MCP broker sets up AS-2, startup gives the MCP broker the identity broker as its call authorizer, the agent credential holder the server registers agents into, the gateway entry's servers, and a streaming in-process client for the embedded vMCP that serves only its endpoint. Startup fails if the gateway endpoint is not AS-2's resource. Production configures no gateway entry, so it is unchanged. The mecak8s slice test runs a session's tool call on the gateway route: after approval, AS-1 issues an ID-JAG for the call's operation and the embedded vMCP accepts the gateway token; a call approved after the broker session closed is refused with session_closed, without an ID-JAG or a reopen. The MCP authorization fixture now gives its late identity broker a credential holder. Co-authored-by: Mecatl <noreply@mecatl.dev>
… a changed call The send-once record is keyed by root session as well, holds at most 4096 claims per root session, and fails closed for that root only (code send_once_full). Deleting a session forgets its claims; closing or losing a lease keeps them, as the session can resume. A 401 counts as invalid_token only when a parsed Bearer challenge has error="invalid_token", so a quoted value can no longer mark a call as never run. A tools/call refused as changed after authorization is a bug, not a reason to authorize again: it is not sent and maps to internal. The embedded gateway shortcut now names #2812. Co-authored-by: Mecatl <noreply@mecatl.dev>
…fe holder The argument hash canonicalizes with encoding/json/jsontext, whose strict decoding also refuses invalid UTF-8, lone surrogate escapes, and duplicate names; only the I-JSON number range and the object-only top level are still checked here. Every vector and refusal is unchanged. Gateway loses the unused Resource, and the embedded-only HTTP client factory moves from GatewayCalls to ToolHiveConfig.EmbeddedGatewayClient. The process exposes the installed gateway endpoint, which startup checks against the AS-2 resource without a callback side effect. agentcredential.Holder Drop and Close are safe on a nil Holder, so the server drops its nil guards. Co-authored-by: Mecatl <noreply@mecatl.dev>
…redirects A table covers every row of outcomeForAuthorization and outcomeForAttempts, with a broker-level no_grant case that dispatches nothing. A 401 without error="invalid_token" on the tools/call is an unknown outcome kept as sent and not retried; on the handshake it is not sent. A redirect on initialize is never followed. The mecak8s slice reads each call operation from its gateway_call diagnostic instead of deriving it, and scans the transcript, events, and diagnostics of a real gateway tool call for any JWT, with a planted positive control. Its stand-in table names #2812 for the external gateway, and the app leak test comment no longer says tool content cannot carry tokens. Co-authored-by: Mecatl <noreply@mecatl.dev>
…tcuts Co-authored-by: Mecatl <noreply@mecatl.dev>
…m in the send-once record Co-authored-by: Mecatl <noreply@mecatl.dev>
A tools/call reply over 8 MiB was never seen by the reply watcher, so a call that completed was reported as an unknown outcome, its result dropped, and its send-once claim held until the session was deleted. The SDK returns a result only from the call's own reply, so a returned result now completes the attempt. Also mark the Gateway interface's missing vMCP resource as a shortcut removed by #2728, and state the #2537 connect-link decision. Co-authored-by: Mecatl <noreply@mecatl.dev>
…outcome A JSON-RPC error reply to a sent tools/call counted as a completed call, so the model was told the call failed even when the error (such as -32603) can come after the tool ran, inviting a second run. Only the JSON-RPC errors that reject the request itself (-32700, -32600, -32601, -32602) now complete the call; any other is an unknown outcome. The unknown-outcome text no longer claims no answer arrived. The declined-requests shortcut now names #2821. Co-authored-by: Mecatl <noreply@mecatl.dev>
… removed by #2822 Co-authored-by: Mecatl <noreply@mecatl.dev>
Co-authored-by: Mecatl <noreply@mecatl.dev>
… call A refusal the outcome reports only as internal, such as AS-1's ID-JAG issuance reasons (sub_id_missing and others from ToolHive v0.66.0), left no trace of its cause: the gateway_call diagnostic carried only the outcome code. It now adds the broker error's code, reason, and detail, which never holds credentials. The model-visible result is unchanged. Co-authored-by: Mecatl <noreply@mecatl.dev>
ChrisJBurns
left a comment
There was a problem hiding this comment.
-
medium · security — Renewal-time identity anomalies (another agent, expired credential, broker refusal) are swallowed without any diagnostic (
internal/agentcredential/holder.go:4766)In
renew, a failure fromRegisterAgentorcheckRegistration(includingErrOtherAgent, where the broker returned a different SPIFFE identity) is dropped. If the held credential has not expired yet, the caller gets it back and nothing is logged. On the run-entry path the server logs "identity broker returned another agent" at WARN (identity_broker.go), but the gateway-call renewal path has no logger, andgatewayCall.logrecords only the outcome code. A broker that misbehaves or is compromised during renewal therefore leaves no trace until the credential expires.Recommendation: Give the Holder a diagnostics sink, or return the renewal error alongside the fallback credential, so the MCP broker can log ErrOtherAgent and ErrExpired at WARN. This matches the run-entry path. At minimum, separate ErrOtherAgent from transient broker errors and surface it.
-
medium · maintainability — PR mixes a mechanical credential-holder extraction, a new streaming transport, and the gateway feature in one ~6,600-line diff
The diff does three separable things. It moves the server's
identityAgentsmap and renewal logic into a newinternal/agentcredentialpackage, which changesidentity_broker.go,service.goand the internal tests. It addsinprocesshttp.NewStreamingClientwith its own panic, abort and cancel semantics. It also adds the gateway sending path (gatewaycall.go,gateway_call.go, send-once record, argument hash, outcome table). A cold reviewer has to keep all three in mind at once, and the behaviour-preserving refactor ofregisterIdentityAgentis hidden among roughly 4,000 lines of new feRecommendation: Split into stacked PRs: (1) extract
agentcredential.Holderfrom the server with no behaviour change, (2) addNewStreamingClient, (3) add the gateway path. If the PR stays as one, give the description a "where to focus" map (gatewaycall.go classify/RoundTrip, gateway_call.go run, holder.go Ensure/Credential) and mark which hunks are pure moves.
| func (g *gatewayCall) run(ctx context.Context) (session.ToolResult, string, error) { | ||
| fail := func(o gatewayOutcome) (session.ToolResult, string, error) { return o.result(g.call.ID), o.code, nil } | ||
| refused := func(err error) gatewayOutcome { | ||
| errors.As(err, &g.authErr) |
There was a problem hiding this comment.
medium · correctness — The new diagnostic captures only identity-broker errors, so the other "internal" failures are still logged without a cause
The refused closure records the error only when errors.As(err, &g.authErr) matches *identitybroker.Error, and log adds detail only in that case. Several other failures still map to outcomeInternal with nothing in the log. These include the plain errors.New("mcpbroker: authorization carries no gateway token or attempt") at line 179, non-broker transport errors from AuthorizeCall or Credentials.Credential, and SendOnce store errors from Claim that are not sentinels. The Route error at line 108 is also dropped: it returns fail(outcomeInternal) directly and never reaches `ref
Recommendation: Keep the terminal error itself on gatewayCall (e.g. g.err error), not just a *identitybroker.Error. In log, emit "error", g.err.Error() for every non-nil error, and keep the broker_code/reason/detail fields when errors.As matches. Send the Route failure at line 108 through the same capture. The broker's Error() never carries credentials, and the local errors are fixed strings, so the no-token/no-arguments guarantee still holds.
There was a problem hiding this comment.
Fixed in 3380fae: a failure before a send that isn't the broker's (no, dropped, expired or other-agent credential; send-once; routing; deadline) is logged as cause, from fixed text only, since an argument error could quote the arguments.
|
|
||
| func TestGatewayTokenNeverLeaks(t *testing.T) { | ||
| const canary = "LEAKCANARY" | ||
| paths := map[string]func(*gatewayFixture){ |
There was a problem hiding this comment.
medium · maintainability — The token leak test does not cover the new broker-refusal diagnostic fields
This commit makes log copy identitybroker.Error.Detail into diagnostics as broker_detail (gateway_call.go:271). The doc comment says the detail "never holds credentials", but no test checks that. TestGatewayTokenNeverLeaks covers only the allow, 401 retry, unknown, URL-elicitation and elicit-during paths. None of them has a broker refusal, so f.diag.String() is never checked for the canary while the broker_* fields are present.
Recommendation: Add a "broker refusal" case to the paths map in TestGatewayTokenNeverLeaks, for example f.auth.script = []error{identitybroker.NewError(identitybroker.CodeInternal, "", "...")}. The case needs its own assertions, because the current positive control requires something to have been sent. Alternatively, add a test where a refusal follows a 401 retry, so a gateway token was already minted, and check that the canary never appears in the diagnostics.
There was a problem hiding this comment.
Fixed in 3380fae: the leak test has a "refused on retry" path, where a gateway token was minted and then the broker refused, so the broker_* fields are checked for the canary.
| } | ||
|
|
||
| // Shortcut: each gateway call uses its own vMCP session, and a 404 for the session fails the call; removed by #2564 (vMCP session reuse and a new session on 404). | ||
| cs, err := client.Connect(ctx, transport, &mcpsdk.ClientSessionOptions{ProtocolVersion: gatewayProtocolVersion}) |
There was a problem hiding this comment.
high · correctness — Gateway initialization has no bounded handshake timeout
The new path calls client.Connect with the session operation context, while gatewayInProcessClient explicitly supplies no HTTP timeout. Ordinary interactive runs need not have a deadline, so a stalled initialize handler can hold the tool execution and send-once claim indefinitely until someone cancels the run; the existing MCP connection path instead applies a default connection timeout.
Recommendation: Apply a bounded initialization context, following the existing MCP connect-timeout convention, without imposing that short handshake deadline on the subsequent tool execution. Add a test whose initialize handler waits for context cancellation.
There was a problem hiding this comment.
Fixed in 3380fae: the handshake now ends at the authorization's send deadline (SendBy), and the attempt is not sent. The SDK detaches the session from the connect context, so the tool call itself is not bounded by it. Test: TestSendGatewayCallHandshakeEndsAtSendBy. Without the fix, cancelling the run still ended a hung handshake and released the claim, so this was a liveness issue, not a correctness one.
| h.mu.Unlock() | ||
| return e.credential(), nil | ||
| } | ||
| if f := h.renewals[key]; f != nil { |
There was a problem hiding this comment.
medium · correctness — A replacement credential can join the previous root's renewal
Renewal deduplication uses only Key, whereas Drop invalidates the root epoch without removing or distinguishing its in-flight renewal. If run entry registers a replacement root while the old renewal remains in flight, a forced renewal—or a later lookup past the replacement credential's renewal threshold—waits on that obsolete flight and receives its ErrDropped instead of renewing the current credential.
Recommendation: Associate renewal flights with the root epoch and join only a flight belonging to the current root. Ensure an old flight's cleanup cannot remove a replacement flight, and add a blocked-renewal test covering Drop, re-registration, and renewal of the replacement. --- ## No Issues
There was a problem hiding this comment.
True, but the only effect is a spurious identity_unavailable: a renewal never returns another root's credential, and it needs a close and reopen during a renewing call. Deferred to #2562, which rewrites renewal; the requirement is now on that ticket.
| if current := r.entries[key.AgentSessionID]; current != e { | ||
| return current.credential(), nil | ||
| } | ||
| if err != nil { |
There was a problem hiding this comment.
medium · correctness — A forced renewal after svid_expired swallows the renewal error and returns the credential the broker just rejected
renew returns the held credential with a nil error whenever RegisterAgent fails and e.reg.ExpiresAt is still in the future by the local clock. That also covers renew=true, the path gatewayCall.authorize takes after AuthorizeCall returned svid_expired for that same credential. The second AuthorizeCall then fails with svid_expired again, and the call is reported as svid_expired ("could not be renewed"). The real cause, such as a retryable CodeUnavailable from the broker or a context deadline, is lost, and nothing logs it.
Recommendation: When renew was forced, return the renewal error instead of the held credential. Alternatively, return both the held credential and a wrapped error so authorize can map it, for example to outcomeUnavailable. At minimum, give the caller the renewal error so the MCP broker can log it.
There was a problem hiding this comment.
| } | ||
| } | ||
|
|
||
| func TestGatewayCallRetriesOneRejectedToken(t *testing.T) { |
There was a problem hiding this comment.
medium · maintainability — Rejected-token retry test does not verify token replacement
TestGatewayCallRetriesOneRejectedToken checks reauthorization and the new attempt ID, but never checks the retry’s Authorization header. Because RejectNextToken rejects one request regardless of its token and then accepts any bearer value, the test would still pass if the retry used the rejected gateway token with the new attempt ID.
Recommendation: Assert from f.wire.requests() that requests carrying att_1 use gateway-token-1 and requests carrying att_2 use gateway-token-2, including each attempt’s handshake. Alternatively, configure the gateway verifier to keep refusing the first token.
There was a problem hiding this comment.
Fixed in 3380fae: the test asserts each attempt's messages carry that attempt's own token, including the handshake.
| } | ||
| } | ||
|
|
||
| func TestSendOnceRefusesSecondSendIncludingAcrossRetry(t *testing.T) { |
There was a problem hiding this comment.
medium · maintainability — Send-once tests omit competing in-flight executions
The new send-once tests exercise duplicate claims and duplicate Execute calls only sequentially, after the first operation has returned. They do not verify the critical in-flight guarantee: two executions of the same operation must not both reach the gateway, including while the winning execution is waiting for fresh authorization after invalid_token.
Recommendation: Add a channel-coordinated test that blocks the first execution after claiming, runs a competing execution with the same call ID and arguments, and asserts already_sent with exactly one dispatched call. Include a variant that blocks reauthorization between the first rejected attempt and its retry. --- ## No Issues
There was a problem hiding this comment.
The code is correct (Claim refuses while the key is held, and the claim is held through the retry). The concurrency test is now a requirement on #2730, where the durable, multi-replica record needs it.
|
|
||
| func (g *gatewayCall) release(ctx context.Context) { | ||
| if g.claim { | ||
| _ = g.r.gatewayCalls.SendOnce.Release(context.WithoutCancel(ctx), g.scope.root, g.op, g.hash) |
There was a problem hiding this comment.
medium · maintainability — Send-once state transition errors are silently discarded
Both release and complete discard errors returned by SendOnceRecord, and release clears its local claim flag even if releasing failed. Although the current in-memory implementation always succeeds, the interface explicitly permits failures; silently ignoring them obscures the lifecycle contract and makes a replacement implementation difficult to diagnose safely.
Recommendation: Handle both errors explicitly, preserving the completed tool result while emitting a token-free diagnostic for failed state transitions. Document whether a failed release retains the claim, and add a failing-record test that demonstrates the intended behavior.
There was a problem hiding this comment.
Leaving as is: the in-memory record never returns an error, and a durable record's failure would fail closed. A failed Release keeps the claim (already_sent later), and a failed Complete only means the claim is never evicted. Neither can cause a resend.
| } | ||
| // A returned error never carries a token either. | ||
| f := newGatewayFixture(t, gatewayFixtureOptions{tokenPrefix: "gw-" + canary + "-", agentToken: "agent-" + canary}) | ||
| ctx, cancel := context.WithCancel(mainAgent(t)) |
There was a problem hiding this comment.
medium · maintainability — Broker-level cancellation during a gateway call is never tested deterministically
The only broker-level cancellation case is the pre-cancelled context at the end of TestGatewayTokenNeverLeaks. With a pre-cancelled context, beginOperation's AfterFunc cancels opCtx in another goroutine, so Execute may return at opCtx.Err() (runtime.go:665) before executeGateway runs. The test passes either way. Nothing covers a cancel after the tools/call was written: the ctx.Err() return at gateway_call.go:85, the claim being kept for an Unknown attempt, and a later Execute of the same call returning already_sent.
Recommendation: Add a test that cancels mid-call, for example with f.g.ElicitDuringCall plus a wire hook that cancels the context once the tools/call has been sent. Assert that Execute returns context.Canceled, that the diagnostic code is outcome_unknown, and that f.claim(t, op) returns ErrAlreadySent. Add the matching case with a cancel before the tools/call is written, and assert the claim is released.
There was a problem hiding this comment.
The code is correct (an unknown attempt keeps its claim, and the call returns ctx.Err()). #2731 changes what a cancelled call returns, so the deterministic test is now a requirement there.
| if last.Rejection == mcpadapter.GatewayRejectionInvalidToken { | ||
| // One retry with a fresh token, reporting the rejected attempt with | ||
| // the call's own fields. | ||
| if auth, err = g.authorize(ctx, last.AttemptID); err != nil { |
There was a problem hiding this comment.
medium · maintainability — Authorization failing on the invalid_token retry is untested
If the second AuthorizeCall after a 401 invalid_token fails (denied, svid_expired, unavailable), run returns outcomeForAuthorization(err). The deferred releaseUnlessSent then has to release the claim, because the only attempt was refused. Every retry test scripts the second authorization to succeed, so neither the outcome mapping nor the claim release on this send-once path is checked.
Recommendation: Add a case with f.wire.setOnToolsCall(f.g.RejectNextToken) and f.auth.script = []error{nil, identitybroker.ReasonError(identitybroker.ReasonDenied, "")}. Assert the denied row, two AuthorizeCall requests with the second carrying RejectedAttemptID att_1, and that f.claim(t, op) succeeds (claim released).
There was a problem hiding this comment.
Added in 3380fae: TestGatewayCallRefusedOnTheRetryReleasesTheClaim checks the denied row, the retry reporting att_1, and that the claim is released.
- The gateway handshake ends at the authorization's send deadline, so a hung gateway cannot hold a tool call open; the attempt is not sent. - A renewal after svid_expired that fails returns its error instead of the credential the broker just refused, so the call reports the real cause. A lookup past half-life still gets the unexpired held credential. - The gateway_call diagnostic names a failure before a send that is not the broker's (another agent, an expired or missing credential, send-once, routing) and the last attempt's HTTP status. - The event-stream reply watcher, like the SDK, counts a last event left unterminated at EOF. - Comments: classify's default case, Complete, and GatewayCalls.Servers. - Tests: each retry attempt carries its own token; a refusal on the retry releases the claim; the leak test covers a broker refusal; the streaming reply test can fail; a registration dropped in flight; the stream parser. Co-authored-by: Mecatl <noreply@mecatl.dev>
|
Replying to the two items in the review summary:
|
Closes #2536
Summary
With the identity broker on and a gateway entry configured, the MCP broker now authorizes every tool call on a gateway route with the identity broker and sends it to the gateway with that call's own token, instead of with its per-session OAuth grant. Production configures no gateway entry until #2533, so production behaviour is unchanged.
internal/adapter/mcpbroker/argumenthash.go): RFC 8785 viaencoding/json/jsontextCanonicalize, refusing arguments outside I-JSON (non-finite numbers, integers beyond ±2^53, invalid UTF-8, lone surrogates, duplicate keys) before authorizing. Vectors intestdata/argumenthash/vectors.jsonfor Bind each gateway token to its call #2727 to share.internal/agentcredential): one holder shared by the server (run entry) and the MCP broker (lookup). The lookup never registers or reopens; renewal past half-life and aftersvid_expiredis single-flight; a dropped session's entry cannot come back.internal/adapter/mcp/gatewaycall.go): a go-sdk session per attempt with multi-round-trip disabled, protocol pinned to 2025-11-25, no redirects, and no reconnects. The token and_meta(dev.mecatl/operation,dev.mecatl/attempt) go on every message; the call is refused before writing if it differs from what was authorized. Each attempt is not sent, refused (401invalid_token, 404 session, -32042: design step (9)), completed (a result, or a JSON-RPC error that rejects the request itself: -32700, -32600, -32601, -32602), or unknown (no answer, another HTTP error, or another JSON-RPC error such as -32603). Requests the gateway sends during a call are declined.internal/adapter/mcpbroker/gateway*.go,sendonce.go):AuthorizeCall→ claim send-once (operation ID, argument hash, per root session) → send → one retry after a rejected token withRejectedAttemptID→ outcome table. Gateway routes no longer ask for the MCP broker's OAuth at call time.internal/app,internal/identitybroker/adapter/inprocesshttp): the embedded vMCP is reached through a streaming in-process client; the identity broker, the credential holder, and the gateway servers are handed to the bundled MCP broker.Design: steps (6), (8), (9) (including which answers mean a call never ran, #2804), "Pauses and restart".
Shortcuts
Each is marked in code with
// Shortcut: …; removed by #NNNNand listed in #2536:send_once_full; cleared when the session is deletedGatewayGatewaycarries no vMCP resource; startup checks the embedded endpoint is AS-2's resourcemcp_owner_action_requiredconsent_requiredhas no approval linkDecisions made while building: a sub-agent or team member without a credential gets
identity_unavailable; a request the gateway sends during a call is declined and the tool's real answer is returned with a note (the call counts as sent); a 401 withoutinvalid_tokenafter the call was written is an unknown outcome; a request that differs on the wire from what was authorized is refused asinternaland not authorized again (it can only happen through a bug); a result too large for the reply watcher (over 8 MiB) still completes the call when the SDK returns it;_metaon error replies goes inerror.data._meta.Testing
identityharnessgateway stand-in (a separate HTTP server: sessions, 404, elicitation during a call, dropped connection after write, redirect, 401 with and withoutinvalid_token); the outcome table row by row; send-once per session; a leak test with a positive control.cmd/mecak8sTestIdentityBrokerSessionToolCallThroughGatewaydrives a real session's tool call through startup wiring: AS-1 issues one ID-JAG for the call's operation, the embedded vMCP accepts the token, a closed session is refused without reopening, and no JWT appears in the transcript, events, or diagnostics.golangci-lint0 issues; race tests for mcpbroker, mcp, agentcredential, identitybroker, identityharness, server identity, app, and mecak8s;go run ./cmd/mecademoshows the tool call, permission ask and approval, and result.Found while building, for #2537: with no provider credential and no other backend, the embedded vMCP ends the session after
initialize, so until #2537 a gateway call fails asgateway_session_not_found. The slice test adds an anonymous offline backend to avoid this.Design: a 5xx after the request was written is an unknown outcome (#2819); which JSON-RPC errors mean the call did not run, and the 5xx failure row (#2820).
API compatibility
Assessment: compatible; no major bump, no feature identifier. The new behaviour is reachable only with the identity broker on and a gateway entry, which only the test-only
IdentityBrokerGatewayServersconfigures until #2533. There, tool calls on gateway routes return new model-visible tool-error texts (codesidentity_unavailable,no_grant, the closing reasons,denied,consent_required,svid_expired,unavailable,internal,refused_arguments,connection_required,gateway_session_not_found,token_rejected,already_sent,send_once_full,outcome_unknown) inside the existing tool-result and event schema, and gateway routes no longer show the MCP broker's OAuth prompt at call time.mcp_authorization_pendingis not produced or changed. One change is reachable in production: if a session is closed or deleted while its agent's registration is in flight, run entry now returns the existingErrIdentityBrokerUnavailableinstead of storing a credential for the closed session. No wire schema, request-level error, or stored data changes. Upgrade implications: none.