Skip to content

feat(gateway): operator resolution path for delivery_uncertain receipts (SEL-351 follow-up) - #393

Open
PetrefiedThunder wants to merge 2 commits into
mainfrom
codex/operator-receipt-resolution
Open

PetrefiedThunder wants to merge 2 commits into
mainfrom
codex/operator-receipt-resolution

Conversation

@PetrefiedThunder

@PetrefiedThunder PetrefiedThunder commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

A signed delivery_uncertain receipt previously had no operator resolution caller. Add a bootstrap-admin-only POST /v1/admin/receipts/{receipt_id}/resolution and a thin gateway/scripts/amw_resolutions.py client. The original receipt, signature, debit and agent replay remain unchanged; readers discover the separate resolution through the existing receipt endpoint and wallet chain.

Resolution v0.2 signs the SHA-256 digest of canonical evidence capped at 16 KiB. Exact outcome/evidence replays return the same row; conflicting evidence or outcomes return 409. Refund intent is part of the signed evidence. Explicit confirmed non-delivery refunds use the existing money service's separate refund-{ledger_entry_id} entry, then its durable dispatch compensation checkpoints and once-only permit budget release. Retries converge after interruption before the refund or before budget release.

An explicitly registered RefundFixtureStatusLookup reads only the fixture /status contract with bearer authentication, a five-second timeout, bounded response bytes, disabled redirects, and no retained raw response. Nothing is registered by default. Executed answers support success; unfenced negative answers, pending/unknown answers, errors and malformed responses remain inconclusive and write no resolution or refund.

Related issues

SEL-351 follow-up, replacing the unused append_resolution TODO with an operator-only path.

Area and license

  • Gateway changes are under FSL-1.1-ALv2; Python SDK changes are Apache-2.0.
  • No proprietary hosted-service code or secrets are included.

Spec changes

  • No files in spec/ or normative protocol material are changed.

Migration and compatibility

058_resolution_evidence follows 057_receipt_permit_seq_unique, adding nullable canonical evidence and evidence_hash columns. Existing v0.1 rows retain their original bytes and remain verifiable offline. New operator records use v0.2; the SDK verifies its evidence digest as well as the signature and original-receipt link. The existing v0.1 golden signing snapshot stays unchanged, and CHANGELOG records the v0.2 version bump.

The rollout document covers image/head compatibility and rollback. Downgrade refuses while signed evidence exists, rather than discarding it. No staging/production migration or deployment was performed.

Security

Reuse AuthContext.require_bootstrap_admin() without adding an authentication scheme. Agent/wallet credentials, demo credentials and permits cannot authorize the route. The operator route is absent from MCP tools and agent discovery. Every handled resolution attempt records a control-plane audit event with an operator credential fingerprint or key id, including conflicts and inconclusive lookups. The adapter never logs its token or persists provider response bodies. Operator evidence is visible to authorized receipt readers, so the runbook requires review/redaction of free text and provider-specific fields in addition to automatic credential-field redaction.

A terminal dispatch row does not fence a paused sender. Negative lookup answers for a historical dispatch claim therefore remain inconclusive. Manual non-delivery evidence requires the operator to establish that senders have stopped. Successful resolutions never refund, and refund intent cannot be changed after append.

Validation

Local tests used fakes/MockTransport and disposable PostgreSQL 16 at 127.0.0.1:55439. All databases created for this change were dropped afterward. Cached Python 3.14.9, pytest 9.2.0, Ruff 0.17.3 and mypy 2.4.1 were used without package-network access.

Check Passed Failed Skipped
Full gateway, uv run --no-project --active python -m pytest -n 6 --timeout 300 -q 6625 0 107
PostgreSQL migration round-trip 1 0 0
PostgreSQL operator race and existing receipt-resolution chain proof 2 0 0
Existing tests/test_mcp_postgres_multiprocess.py 17 0 0
Python SDK tests 474 0 0
Site Python verifier tests 5 0 0
Site Node tests 42 0 0
Verifier vectors 96 0 0

The plain gateway run skips opt-in tests; the requested PostgreSQL proofs above were run separately with the CI opt-in flags, ENVIRONMENT=test, and disposable database acknowledgement. Multiprocess tests additionally used STATE_BACKEND=postgres and MCP_STRESS_DB_ISOLATED=1.

Ruff check/format passed for every Python project returned by CI project discovery. Pinned mypy passed for all 215 application source files. OpenAPI export matches the app. SDK wheel and source distribution built successfully. Staged secret scans, private-key detection, whitespace, merge-conflict, Ruff and formatting hooks passed. The isolated mypy pre-commit hook was skipped because its resolver could not obtain mypy from the offline package cache; the direct pinned application check passed. An all-files pre-commit run also reported existing issues in untouched files; those were left outside this PR.

New coverage includes actual agent-key refusal, evidence size bounds and tampering, stable v0.1 verification, conflicting re-resolution, fixture lookup failures/timeouts, separate exactly-once refunds, interruption before refund and after refund/before budget release, unchanged original receipt bytes, a fresh-app/fresh-singleton lost-response replay with one dispatch and one debit, offline resolution verification, and concurrent PostgreSQL operator requests.

Open questions

  • Deployment startup must supply and register the fixture origin/token; no deployment wiring or real upstream call was exercised here.
  • Automatically accepting not_executed for a previously claimed send requires a genuine upstream fencing contract. Until one exists, that answer remains inconclusive and the operator must obtain authoritative evidence and stop potential senders before making a manual assertion.

Contributor checklist

  • Added tests for behavior changes and updated relevant documentation.
  • Signed off the commit under the DCO and used the applicable area licenses.
  • Ready for review; no merge or deployment performed.

Note

High Risk
Touches operator auth, signed ledger refunds, and permit budget reconciliation for ambiguous delivery receipts; mistakes could incorrectly release charges or double-refund without careful idempotency.

Overview
Adds a bootstrap-admin-only path to adjudicate signed delivery_uncertain receipts without mutating the original receipt, replay, or redispatching tools.

POST /v1/admin/receipts/{receipt_id}/resolution accepts bounded operator evidence and appends a signed resolution-v0.2 record (migration 058 adds evidence / evidence_hash; v0.1 remains verifiable). Conflicting outcome or evidence returns 409; identical requests replay the same row. refund_credits on confirmed_not_delivered creates a separate ledger refund, with retries finishing interrupted compensation and permit budget release only after a matching refund entry exists.

An optional startup-registered fixture status lookup can corroborate outcomes under strict timeouts and refuses unfenced negative answers when a dispatch claim may still send. Documentation, OpenAPI, CLI amw_resolutions.py, and the Python SDK verifier are updated for v0.2 evidence-hash checks.

Reviewed by Cursor Bugbot for commit 2cfb9ea. Bugbot is set up for automated code reviews on this repo. Configure here.

Follow-up review fixes

  • Receipts without an MCP dispatch attempt (including AWI HTTP uncertain receipts) reject refund_credits with 409 resolution_refund_unsupported before signed refund intent is appended. This chooses the supported-kind rejection option rather than adding a new reservation release marker.
  • Permit reconciliation excludes operator-confirmed non-deliveries only after the exact matching refund ledger entry exists. Signed intent alone continues to count as consumed.
  • Receipt lookup registration, lookup selection, and the fixture adapter share origin normalization for trailing slashes, scheme/host case, and default ports.
  • Dispatch context conflicts return audited 409 resolution_inconclusive without appending a resolution or refunding.

Regression tests cover all four findings, including expired permit reconciliation after completed and interrupted refunds.

Validation of the follow-up worktree changes: focused suites 360 passed, 23 skipped; full gateway suite with -n 6 --timeout 300 6631 passed, 107 skipped; disposable PostgreSQL operator race 1 passed. Ruff check and format pass across all discovered Python projects; pinned mypy 2.4.1 passes all 215 gateway source files.

These follow-up changes are commit 2cfb9ea0 on this branch (review findings: no-attempt refund rejected before signed intent; budget reconciliation excludes durably refunded operator non-deliveries; lookup origin normalization; audited 409 for dispatch-context conflicts).

Signed-off-by: CS <cs@CSs-MacBook-Pro.local>
@vercel

vercel Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
agent-middleware-web Ready Ready Preview Nov 15, 2026 10:20am UTC

Request Review

@coderabbitai

coderabbitai Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 51 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 75278a8 and 2cfb9ea.

Files selected for processing: 23

  • CHANGELOG.md
  • gateway/TROUBLESHOOTING.md
  • gateway/app/db/models.py
  • gateway/app/main.py
  • gateway/app/routers/receipts.py
  • gateway/app/schemas/resolutions.py
  • gateway/app/services/mcp_dispatch_attempts.py
  • gateway/app/services/operator_resolution.py
  • gateway/app/services/permits.py
  • gateway/app/services/receipt_resolutions.py
  • gateway/docs/failure-semantics.md
  • gateway/docs/openapi.json
  • gateway/docs/operator-resolution-runbook.md
  • gateway/docs/partner-first-tool-runbook.md
  • gateway/docs/schema-058-rollout.md
  • gateway/migrations/versions/058_resolution_evidence.py
  • gateway/scripts/amw_resolutions.py
  • gateway/tests/test_amw_resolutions_cli.py
  • gateway/tests/test_operator_resolution.py
  • gateway/tests/test_repair_api_lineage_migration.py
  • gateway/tests/test_resolution_evidence_migration.py
  • sdk/python/src/b2a_sdk/resolution_verifier.py
  • sdk/python/tests/test_resolution_verifier.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-14T20:03:49.135656Z b7af94f PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b7af94fa2a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +260 to +262
if attempt:

async def finish_compensation() -> None:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Release refunded reservations for receipts without dispatch attempts

For an AWI HTTP delivery_uncertain receipt, a manual confirmed_not_delivered resolution with refund_credits=true refunds the wallet and returns success, but this guard skips permit compensation entirely. awi_http_governance.py reserves permit budget before charging and creates receipts without an MCP dispatch attempt, so these receipts legitimately reach this branch with attempt=None. The reservation therefore remains in spent_credits, potentially continuing to deny calls with permit_budget_exceeded after the refund; replaying the resolution cannot repair it. Add a durable, once-only reservation release for this receipt path, or reject unsupported refunds before appending their signed intent.

AGENTS.md reference: gateway/app/services/AGENTS.md:L7-L16

Useful? React with 👍 / 👎.

Comment on lines +267 to +270
await get_permit_service().release_dispatch_budget_once(
attempt.attempt_id,
caller=CallerIdentity.for_wallet(receipt.wallet_id),
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Account for operator refunds in budget reconciliation

After this release successfully compensates an uncertain dispatch, PermitService._consumed_credits() still unconditionally counts the original delivery_uncertain receipt as consumed and never checks its resolution or refund. Once the permit expires or is revoked and becomes idle, the existing reconcile_budgets() background task writes that inflated receipt total back into spent_credits, undoing the completed budget release while budget_released_at prevents another release. Update the reconciliation source of truth to exclude operator-resolved non-deliveries whose refund is durably complete, without excluding records that only contain signed refund intent.

AGENTS.md reference: gateway/app/services/AGENTS.md:L7-L16

Useful? React with 👍 / 👎.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit b7af94f. Configure here.

Written by the Codex CLI engine. Reject refunds for receipts without a dispatch attempt before signing intent; exclude durably refunded operator non-deliveries from budget reconciliation; normalize lookup origins at register and lookup; return audited 409 resolution_inconclusive for dispatch-context conflicts.

Signed-off-by: Christopher <131207452+PetrefiedThunder@users.noreply.github.com>

This branch was successfully deployed

1 active deployment
Preview — 2cfb9ea0 Deployed Nov 15, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants