Skip to content

feat(mcpbroker): authorize gateway calls with the identity broker and send them with a per-call token - #2818

Open
ChrisJBurns wants to merge 18 commits into
mainfrom
feat/mcpbroker-gateway-calls
Open

ChrisJBurns wants to merge 18 commits into
mainfrom
feat/mcpbroker-gateway-calls

Conversation

@ChrisJBurns

@ChrisJBurns ChrisJBurns commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Closes #2536

Summary

With the identity broker on and a gateway entry configured, the MCP broker now authorizes every tool call on a gateway route with the identity broker and sends it to the gateway with that call's own token, instead of with its per-session OAuth grant. Production configures no gateway entry until #2533, so production behaviour is unchanged.

  • Argument hash (internal/adapter/mcpbroker/argumenthash.go): RFC 8785 via encoding/json/jsontext Canonicalize, refusing arguments outside I-JSON (non-finite numbers, integers beyond ±2^53, invalid UTF-8, lone surrogates, duplicate keys) before authorizing. Vectors in testdata/argumenthash/vectors.json for Bind each gateway token to its call #2727 to share.
  • Agent credentials (internal/agentcredential): one holder shared by the server (run entry) and the MCP broker (lookup). The lookup never registers or reopens; renewal past half-life and after svid_expired is single-flight; a dropped session's entry cannot come back.
  • Sending one call (internal/adapter/mcp/gatewaycall.go): a go-sdk session per attempt with multi-round-trip disabled, protocol pinned to 2025-11-25, no redirects, and no reconnects. The token and _meta (dev.mecatl/operation, dev.mecatl/attempt) go on every message; the call is refused before writing if it differs from what was authorized. Each attempt is not sent, refused (401 invalid_token, 404 session, -32042: design step (9)), completed (a result, or a JSON-RPC error that rejects the request itself: -32700, -32600, -32601, -32602), or unknown (no answer, another HTTP error, or another JSON-RPC error such as -32603). Requests the gateway sends during a call are declined.
  • The gateway path (internal/adapter/mcpbroker/gateway*.go, sendonce.go): AuthorizeCall → claim send-once (operation ID, argument hash, per root session) → send → one retry after a rejected token with RejectedAttemptID → outcome table. Gateway routes no longer ask for the MCP broker's OAuth at call time.
  • Wiring (internal/app, internal/identitybroker/adapter/inprocesshttp): the embedded vMCP is reached through a streaming in-process client; the identity broker, the credential holder, and the gateway servers are handed to the bundled MCP broker.

Design: steps (6), (8), (9) (including which answers mean a call never ran, #2804), "Pauses and restart".

Shortcuts

Each is marked in code with // Shortcut: …; removed by #NNNN and listed in #2536:

Shortcut Removed by
Only servers the gateway entry routes use the gateway path; production configures none #2533, #2538
Run ID is the session's run serial, run owner the session owner, no delegation #2557
A missing provider connection fails closed with its link #2537, #2592
Requests the gateway sends during a call are declined #2821
A tool error result counts as a completed call, though the vMCP also returns backend failures after it sent the call (timeout, dropped connection) as tool errors #2822
A vMCP session per call; a 404 for the session fails the call #2564
Send-once record in memory, per root session; completed calls make room when the share (4096) is full, otherwise send_once_full; cleared when the session is deleted #2730
Credentials renewed by registering again #2562
Only the embedded gateway implements Gateway #2812
Gateway carries no vMCP resource; startup checks the embedded endpoint is AS-2's resource #2728
Tool listing still uses the MCP broker's OAuth login #2729
Sign-out and revocation reasons fail the call as tool errors; the run does not stop; no mcp_owner_action_required #2568
consent_required has no approval link #2735
No dispatch/outcome report to the broker; a cancelled call reports the cancellation #2731
No run ID in trace headers #2566

Decisions made while building: a sub-agent or team member without a credential gets identity_unavailable; a request the gateway sends during a call is declined and the tool's real answer is returned with a note (the call counts as sent); a 401 without invalid_token after the call was written is an unknown outcome; a request that differs on the wire from what was authorized is refused as internal and not authorized again (it can only happen through a bug); a result too large for the reply watcher (over 8 MiB) still completes the call when the SDK returns it; _meta on error replies goes in error.data._meta.

Testing

  • Unit tests per stage: hash vectors and refusals; holder renewal, single-flight, drop races; the send path against the identityharness gateway stand-in (a separate HTTP server: sessions, 404, elicitation during a call, dropped connection after write, redirect, 401 with and without invalid_token); the outcome table row by row; send-once per session; a leak test with a positive control.
  • Slice: cmd/mecak8s TestIdentityBrokerSessionToolCallThroughGateway drives a real session's tool call through startup wiring: AS-1 issues one ID-JAG for the call's operation, the embedded vMCP accepts the token, a closed session is refused without reopening, and no JWT appears in the transcript, events, or diagnostics.
  • Reviewed by spec, standards, test-adequacy, security, architecture, and reuse reviewers; their verified findings are fixed in this branch.
  • Root golangci-lint 0 issues; race tests for mcpbroker, mcp, agentcredential, identitybroker, identityharness, server identity, app, and mecak8s; go run ./cmd/mecademo shows the tool call, permission ask and approval, and result.

Found while building, for #2537: with no provider credential and no other backend, the embedded vMCP ends the session after initialize, so until #2537 a gateway call fails as gateway_session_not_found. The slice test adds an anonymous offline backend to avoid this.

Design: a 5xx after the request was written is an unknown outcome (#2819); which JSON-RPC errors mean the call did not run, and the 5xx failure row (#2820).

API compatibility

Assessment: compatible; no major bump, no feature identifier. The new behaviour is reachable only with the identity broker on and a gateway entry, which only the test-only IdentityBrokerGatewayServers configures until #2533. There, tool calls on gateway routes return new model-visible tool-error texts (codes identity_unavailable, no_grant, the closing reasons, denied, consent_required, svid_expired, unavailable, internal, refused_arguments, connection_required, gateway_session_not_found, token_rejected, already_sent, send_once_full, outcome_unknown) inside the existing tool-result and event schema, and gateway routes no longer show the MCP broker's OAuth prompt at call time. mcp_authorization_pending is not produced or changed. One change is reachable in production: if a session is closed or deleted while its agent's registration is in flight, run entry now returns the existing ErrIdentityBrokerUnavailable instead of storing a credential for the closed session. No wire schema, request-level error, or stored data changes. Upgrade implications: none.

ChrisJBurns and others added 12 commits November 16, 2026 18:24
…nts outside I-JSON

argumentsSHA256 canonicalizes params.arguments per RFC 8785 using only the
standard library and refuses what the vMCP could hash differently: invalid
UTF-8, lone surrogate escapes, duplicate member names, non-object values,
non-finite numbers, and integers beyond 2^53. Absent or null arguments hash
and are sent as {}.

Co-authored-by: Mecatl <noreply@mecatl.dev>
…MCP broker

Co-authored-by: Mecatl <noreply@mecatl.dev>
…n, and dropped connections

Co-authored-by: Mecatl <noreply@mecatl.dev>
… send them with a per-call token

A tools/call on a broker route whose server is a gateway server is
authorized by the identity broker for the acting agent and sent to the
gateway with a gateway token for that call alone. The argument hash is
checked first; one svid_expired renews the agent credential and asks
again; one invalid_token is retried reporting the rejected attempt; a
call is claimed in a send-once record before it is sent and released only
when no attempt may have run. Refusals and unknown outcomes are
model-visible tool errors with a code. A request the gateway sends during
the call is declined and the call's real answer is returned with a note.
Gateway routes no longer ask for the MCP broker's OAuth authorization at
call time.

Co-authored-by: Mecatl <noreply@mecatl.dev>
NewStreamingClient serves requests from an http.Handler in process and
streams the response. As net/http's server does, the writer buffers: the
response is sent, and RoundTrip returns, at the handler's first Flush,
once 4 KiB is written, or when the handler returns, and each later Flush
sends what was written since. The caller's context, or closing the body,
cancels the handler; a body read after the context ended fails with its
error. A handler panic drops the response.

Co-authored-by: Mecatl <noreply@mecatl.dev>
…way is configured

When the identity broker has a gateway entry and the bundled MCP broker
sets up AS-2, startup gives the MCP broker the identity broker as its call
authorizer, the agent credential holder the server registers agents into,
the gateway entry's servers, and a streaming in-process client for the
embedded vMCP that serves only its endpoint. Startup fails if the gateway
endpoint is not AS-2's resource. Production configures no gateway entry,
so it is unchanged.

The mecak8s slice test runs a session's tool call on the gateway route:
after approval, AS-1 issues an ID-JAG for the call's operation and the
embedded vMCP accepts the gateway token; a call approved after the broker
session closed is refused with session_closed, without an ID-JAG or a
reopen.

The MCP authorization fixture now gives its late identity broker a
credential holder.

Co-authored-by: Mecatl <noreply@mecatl.dev>
… a changed call

The send-once record is keyed by root session as well, holds at most
4096 claims per root session, and fails closed for that root only
(code send_once_full). Deleting a session forgets its claims; closing
or losing a lease keeps them, as the session can resume.

A 401 counts as invalid_token only when a parsed Bearer challenge has
error="invalid_token", so a quoted value can no longer mark a call as
never run.

A tools/call refused as changed after authorization is a bug, not a
reason to authorize again: it is not sent and maps to internal.

The embedded gateway shortcut now names #2812.

Co-authored-by: Mecatl <noreply@mecatl.dev>
…fe holder

The argument hash canonicalizes with encoding/json/jsontext, whose
strict decoding also refuses invalid UTF-8, lone surrogate escapes, and
duplicate names; only the I-JSON number range and the object-only top
level are still checked here. Every vector and refusal is unchanged.

Gateway loses the unused Resource, and the embedded-only HTTP client
factory moves from GatewayCalls to ToolHiveConfig.EmbeddedGatewayClient.
The process exposes the installed gateway endpoint, which startup checks
against the AS-2 resource without a callback side effect.

agentcredential.Holder Drop and Close are safe on a nil Holder, so the
server drops its nil guards.

Co-authored-by: Mecatl <noreply@mecatl.dev>
…redirects

A table covers every row of outcomeForAuthorization and
outcomeForAttempts, with a broker-level no_grant case that dispatches
nothing. A 401 without error="invalid_token" on the tools/call is an
unknown outcome kept as sent and not retried; on the handshake it is not
sent. A redirect on initialize is never followed.

The mecak8s slice reads each call operation from its gateway_call
diagnostic instead of deriving it, and scans the transcript, events, and
diagnostics of a real gateway tool call for any JWT, with a planted
positive control. Its stand-in table names #2812 for the external
gateway, and the app leak test comment no longer says tool content
cannot carry tokens.

Co-authored-by: Mecatl <noreply@mecatl.dev>
…tcuts

Co-authored-by: Mecatl <noreply@mecatl.dev>
…m in the send-once record

Co-authored-by: Mecatl <noreply@mecatl.dev>
A tools/call reply over 8 MiB was never seen by the reply watcher, so a call
that completed was reported as an unknown outcome, its result dropped, and
its send-once claim held until the session was deleted. The SDK returns a
result only from the call's own reply, so a returned result now completes the
attempt.

Also mark the Gateway interface's missing vMCP resource as a shortcut removed
by #2728, and state the #2537 connect-link decision.

Co-authored-by: Mecatl <noreply@mecatl.dev>
ChrisJBurns and others added 4 commits November 16, 2026 21:33
…outcome

A JSON-RPC error reply to a sent tools/call counted as a completed call, so
the model was told the call failed even when the error (such as -32603) can
come after the tool ran, inviting a second run. Only the JSON-RPC errors that
reject the request itself (-32700, -32600, -32601, -32602) now complete the
call; any other is an unknown outcome. The unknown-outcome text no longer
claims no answer arrived.

The declined-requests shortcut now names #2821.

Co-authored-by: Mecatl <noreply@mecatl.dev>
… removed by #2822

Co-authored-by: Mecatl <noreply@mecatl.dev>
Co-authored-by: Mecatl <noreply@mecatl.dev>
… call

A refusal the outcome reports only as internal, such as AS-1's ID-JAG
issuance reasons (sub_id_missing and others from ToolHive v0.66.0), left no
trace of its cause: the gateway_call diagnostic carried only the outcome
code. It now adds the broker error's code, reason, and detail, which never
holds credentials. The model-visible result is unchanged.

Co-authored-by: Mecatl <noreply@mecatl.dev>

@ChrisJBurns ChrisJBurns left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • medium · security — Renewal-time identity anomalies (another agent, expired credential, broker refusal) are swallowed without any diagnostic (internal/agentcredential/holder.go:4766)

    In renew, a failure from RegisterAgent or checkRegistration (including ErrOtherAgent, where the broker returned a different SPIFFE identity) is dropped. If the held credential has not expired yet, the caller gets it back and nothing is logged. On the run-entry path the server logs "identity broker returned another agent" at WARN (identity_broker.go), but the gateway-call renewal path has no logger, and gatewayCall.log records only the outcome code. A broker that misbehaves or is compromised during renewal therefore leaves no trace until the credential expires.

    Recommendation: Give the Holder a diagnostics sink, or return the renewal error alongside the fallback credential, so the MCP broker can log ErrOtherAgent and ErrExpired at WARN. This matches the run-entry path. At minimum, separate ErrOtherAgent from transient broker errors and surface it.

  • medium · maintainability — PR mixes a mechanical credential-holder extraction, a new streaming transport, and the gateway feature in one ~6,600-line diff

    The diff does three separable things. It moves the server's identityAgents map and renewal logic into a new internal/agentcredential package, which changes identity_broker.go, service.go and the internal tests. It adds inprocesshttp.NewStreamingClient with its own panic, abort and cancel semantics. It also adds the gateway sending path (gatewaycall.go, gateway_call.go, send-once record, argument hash, outcome table). A cold reviewer has to keep all three in mind at once, and the behaviour-preserving refactor of registerIdentityAgent is hidden among roughly 4,000 lines of new fe

    Recommendation: Split into stacked PRs: (1) extract agentcredential.Holder from the server with no behaviour change, (2) add NewStreamingClient, (3) add the gateway path. If the PR stays as one, give the description a "where to focus" map (gatewaycall.go classify/RoundTrip, gateway_call.go run, holder.go Ensure/Credential) and mark which hunks are pure moves.

func (g *gatewayCall) run(ctx context.Context) (session.ToolResult, string, error) {
fail := func(o gatewayOutcome) (session.ToolResult, string, error) { return o.result(g.call.ID), o.code, nil }
refused := func(err error) gatewayOutcome {
errors.As(err, &g.authErr)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium · correctness — The new diagnostic captures only identity-broker errors, so the other "internal" failures are still logged without a cause

The refused closure records the error only when errors.As(err, &g.authErr) matches *identitybroker.Error, and log adds detail only in that case. Several other failures still map to outcomeInternal with nothing in the log. These include the plain errors.New("mcpbroker: authorization carries no gateway token or attempt") at line 179, non-broker transport errors from AuthorizeCall or Credentials.Credential, and SendOnce store errors from Claim that are not sentinels. The Route error at line 108 is also dropped: it returns fail(outcomeInternal) directly and never reaches `ref

Recommendation: Keep the terminal error itself on gatewayCall (e.g. g.err error), not just a *identitybroker.Error. In log, emit "error", g.err.Error() for every non-nil error, and keep the broker_code/reason/detail fields when errors.As matches. Send the Route failure at line 108 through the same capture. The broker's Error() never carries credentials, and the local errors are fixed strings, so the no-token/no-arguments guarantee still holds.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 3380fae: a failure before a send that isn't the broker's (no, dropped, expired or other-agent credential; send-once; routing; deadline) is logged as cause, from fixed text only, since an argument error could quote the arguments.


func TestGatewayTokenNeverLeaks(t *testing.T) {
const canary = "LEAKCANARY"
paths := map[string]func(*gatewayFixture){

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium · maintainability — The token leak test does not cover the new broker-refusal diagnostic fields

This commit makes log copy identitybroker.Error.Detail into diagnostics as broker_detail (gateway_call.go:271). The doc comment says the detail "never holds credentials", but no test checks that. TestGatewayTokenNeverLeaks covers only the allow, 401 retry, unknown, URL-elicitation and elicit-during paths. None of them has a broker refusal, so f.diag.String() is never checked for the canary while the broker_* fields are present.

Recommendation: Add a "broker refusal" case to the paths map in TestGatewayTokenNeverLeaks, for example f.auth.script = []error{identitybroker.NewError(identitybroker.CodeInternal, "", "...")}. The case needs its own assertions, because the current positive control requires something to have been sent. Alternatively, add a test where a refusal follows a 401 retry, so a gateway token was already minted, and check that the canary never appears in the diagnostics.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 3380fae: the leak test has a "refused on retry" path, where a gateway token was minted and then the broker refused, so the broker_* fields are checked for the canary.

Comment thread internal/adapter/mcp/gatewaycall.go Outdated
}

// Shortcut: each gateway call uses its own vMCP session, and a 404 for the session fails the call; removed by #2564 (vMCP session reuse and a new session on 404).
cs, err := client.Connect(ctx, transport, &mcpsdk.ClientSessionOptions{ProtocolVersion: gatewayProtocolVersion})

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high · correctness — Gateway initialization has no bounded handshake timeout

The new path calls client.Connect with the session operation context, while gatewayInProcessClient explicitly supplies no HTTP timeout. Ordinary interactive runs need not have a deadline, so a stalled initialize handler can hold the tool execution and send-once claim indefinitely until someone cancels the run; the existing MCP connection path instead applies a default connection timeout.

Recommendation: Apply a bounded initialization context, following the existing MCP connect-timeout convention, without imposing that short handshake deadline on the subsequent tool execution. Add a test whose initialize handler waits for context cancellation.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 3380fae: the handshake now ends at the authorization's send deadline (SendBy), and the attempt is not sent. The SDK detaches the session from the connect context, so the tool call itself is not bounded by it. Test: TestSendGatewayCallHandshakeEndsAtSendBy. Without the fix, cancelling the run still ended a hung handshake and released the claim, so this was a liveness issue, not a correctness one.

h.mu.Unlock()
return e.credential(), nil
}
if f := h.renewals[key]; f != nil {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium · correctness — A replacement credential can join the previous root's renewal

Renewal deduplication uses only Key, whereas Drop invalidates the root epoch without removing or distinguishing its in-flight renewal. If run entry registers a replacement root while the old renewal remains in flight, a forced renewal—or a later lookup past the replacement credential's renewal threshold—waits on that obsolete flight and receives its ErrDropped instead of renewing the current credential.

Recommendation: Associate renewal flights with the root epoch and join only a flight belonging to the current root. Ensure an old flight's cleanup cannot remove a replacement flight, and add a blocked-renewal test covering Drop, re-registration, and renewal of the replacement. --- ## No Issues

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, but the only effect is a spurious identity_unavailable: a renewal never returns another root's credential, and it needs a close and reopen during a renewing call. Deferred to #2562, which rewrites renewal; the requirement is now on that ticket.

if current := r.entries[key.AgentSessionID]; current != e {
return current.credential(), nil
}
if err != nil {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium · correctness — A forced renewal after svid_expired swallows the renewal error and returns the credential the broker just rejected

renew returns the held credential with a nil error whenever RegisterAgent fails and e.reg.ExpiresAt is still in the future by the local clock. That also covers renew=true, the path gatewayCall.authorize takes after AuthorizeCall returned svid_expired for that same credential. The second AuthorizeCall then fails with svid_expired again, and the call is reported as svid_expired ("could not be renewed"). The real cause, such as a retryable CodeUnavailable from the broker or a context deadline, is lost, and nothing logs it.

Recommendation: When renew was forced, return the renewal error instead of the held credential. Alternatively, return both the held credential and a wrapped error so authorize can map it, for example to outcomeUnavailable. At minimum, give the caller the renewal error so the MCP broker can log it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 3380fae: a renewal requested after svid_expired that fails now returns the renewal's error, so the call reports the real cause. A lookup past half-life still gets the unexpired held credential, as the design says. #2562 replaces renewal by registering again.

}
}

func TestGatewayCallRetriesOneRejectedToken(t *testing.T) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium · maintainability — Rejected-token retry test does not verify token replacement

TestGatewayCallRetriesOneRejectedToken checks reauthorization and the new attempt ID, but never checks the retry’s Authorization header. Because RejectNextToken rejects one request regardless of its token and then accepts any bearer value, the test would still pass if the retry used the rejected gateway token with the new attempt ID.

Recommendation: Assert from f.wire.requests() that requests carrying att_1 use gateway-token-1 and requests carrying att_2 use gateway-token-2, including each attempt’s handshake. Alternatively, configure the gateway verifier to keep refusing the first token.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 3380fae: the test asserts each attempt's messages carry that attempt's own token, including the handshake.

}
}

func TestSendOnceRefusesSecondSendIncludingAcrossRetry(t *testing.T) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium · maintainability — Send-once tests omit competing in-flight executions

The new send-once tests exercise duplicate claims and duplicate Execute calls only sequentially, after the first operation has returned. They do not verify the critical in-flight guarantee: two executions of the same operation must not both reach the gateway, including while the winning execution is waiting for fresh authorization after invalid_token.

Recommendation: Add a channel-coordinated test that blocks the first execution after claiming, runs a competing execution with the same call ID and arguments, and asserts already_sent with exactly one dispatched call. Include a variant that blocks reauthorization between the first rejected attempt and its retry. --- ## No Issues

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code is correct (Claim refuses while the key is held, and the claim is held through the retry). The concurrency test is now a requirement on #2730, where the durable, multi-replica record needs it.


func (g *gatewayCall) release(ctx context.Context) {
if g.claim {
_ = g.r.gatewayCalls.SendOnce.Release(context.WithoutCancel(ctx), g.scope.root, g.op, g.hash)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium · maintainability — Send-once state transition errors are silently discarded

Both release and complete discard errors returned by SendOnceRecord, and release clears its local claim flag even if releasing failed. Although the current in-memory implementation always succeeds, the interface explicitly permits failures; silently ignoring them obscures the lifecycle contract and makes a replacement implementation difficult to diagnose safely.

Recommendation: Handle both errors explicitly, preserving the completed tool result while emitting a token-free diagnostic for failed state transitions. Document whether a failed release retains the claim, and add a failing-record test that demonstrates the intended behavior.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving as is: the in-memory record never returns an error, and a durable record's failure would fail closed. A failed Release keeps the claim (already_sent later), and a failed Complete only means the claim is never evicted. Neither can cause a resend.

}
// A returned error never carries a token either.
f := newGatewayFixture(t, gatewayFixtureOptions{tokenPrefix: "gw-" + canary + "-", agentToken: "agent-" + canary})
ctx, cancel := context.WithCancel(mainAgent(t))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium · maintainability — Broker-level cancellation during a gateway call is never tested deterministically

The only broker-level cancellation case is the pre-cancelled context at the end of TestGatewayTokenNeverLeaks. With a pre-cancelled context, beginOperation's AfterFunc cancels opCtx in another goroutine, so Execute may return at opCtx.Err() (runtime.go:665) before executeGateway runs. The test passes either way. Nothing covers a cancel after the tools/call was written: the ctx.Err() return at gateway_call.go:85, the claim being kept for an Unknown attempt, and a later Execute of the same call returning already_sent.

Recommendation: Add a test that cancels mid-call, for example with f.g.ElicitDuringCall plus a wire hook that cancels the context once the tools/call has been sent. Assert that Execute returns context.Canceled, that the diagnostic code is outcome_unknown, and that f.claim(t, op) returns ErrAlreadySent. Add the matching case with a cancel before the tools/call is written, and assert the claim is released.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code is correct (an unknown attempt keeps its claim, and the call returns ctx.Err()). #2731 changes what a cancelled call returns, so the deterministic test is now a requirement there.

if last.Rejection == mcpadapter.GatewayRejectionInvalidToken {
// One retry with a fresh token, reporting the rejected attempt with
// the call's own fields.
if auth, err = g.authorize(ctx, last.AttemptID); err != nil {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium · maintainability — Authorization failing on the invalid_token retry is untested

If the second AuthorizeCall after a 401 invalid_token fails (denied, svid_expired, unavailable), run returns outcomeForAuthorization(err). The deferred releaseUnlessSent then has to release the claim, because the only attempt was refused. Every retry test scripts the second authorization to succeed, so neither the outcome mapping nor the claim release on this send-once path is checked.

Recommendation: Add a case with f.wire.setOnToolsCall(f.g.RejectNextToken) and f.auth.script = []error{nil, identitybroker.ReasonError(identitybroker.ReasonDenied, "")}. Assert the denied row, two AuthorizeCall requests with the second carrying RejectedAttemptID att_1, and that f.claim(t, op) succeeds (claim released).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 3380fae: TestGatewayCallRefusedOnTheRetryReleasesTheClaim checks the denied row, the retry reporting att_1, and that the claim is released.

- The gateway handshake ends at the authorization's send deadline, so a hung
  gateway cannot hold a tool call open; the attempt is not sent.
- A renewal after svid_expired that fails returns its error instead of the
  credential the broker just refused, so the call reports the real cause. A
  lookup past half-life still gets the unexpired held credential.
- The gateway_call diagnostic names a failure before a send that is not the
  broker's (another agent, an expired or missing credential, send-once,
  routing) and the last attempt's HTTP status.
- The event-stream reply watcher, like the SDK, counts a last event left
  unterminated at EOF.
- Comments: classify's default case, Complete, and GatewayCalls.Servers.
- Tests: each retry attempt carries its own token; a refusal on the retry
  releases the claim; the leak test covers a broker refusal; the streaming
  reply test can fail; a registration dropped in flight; the stream parser.

Co-authored-by: Mecatl <noreply@mecatl.dev>
@ChrisJBurns

Copy link
Copy Markdown
Contributor Author

Replying to the two items in the review summary:

  • Renewal anomalies swallowed (security): the wrong-agent or expired registration was never stored or used. The fallback was the held credential, which is for the right agent and stays valid until it expires, as the design says. The gap was visibility, and 3380fae closes it:
    • a renewal requested after svid_expired that fails now returns its error (for example "broker returned another agent") instead of the refused credential;
    • the gateway_call diagnostic logs that cause.
  • Split the PR: keeping it as one. The holder extraction exists because the gateway path needs the credential, and the streaming client exists because the gateway path needs streaming. Where to focus: classify and RoundTrip in internal/adapter/mcp/gatewaycall.go, run in internal/adapter/mcpbroker/gateway_call.go, and Ensure/Credential in internal/agentcredential/holder.go.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MCP broker authorizes gateway calls and sends them to the configured gateway

1 participant