Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,16 @@ This repository starts with a fresh history. Releases are listed here once tagge

## [Unreleased]

### Operator receipt resolution (SEL-351 follow-up)

- Add a bootstrap-admin-only resolution endpoint and operator CLI for signed
delivery_uncertain receipts, with bounded evidence and explicit refunds.
- Add schema 058 evidence columns and resolution-v0.2 evidence-hash signatures;
preserve offline verification of resolution-v0.1 and original receipt bytes.
- Keep original replays unchanged; repeat requests finish interrupted refunds
through separate ledger entries. Optional fixture lookups remain unregistered
by default and refuse unfenced negative answers.

- Receipts: additive signed fields `permit_digest`, `approval_digest`,
`counterparty`, `sequence` (per-permit, from 0 when a caller opts in; left
unset otherwise), and `prev_receipt_digest` (omitted at sequence 0). Digests
Expand Down
2 changes: 1 addition & 1 deletion gateway/TROUBLESHOOTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ If the upstream exposes a status endpoint, a deployment can register an optional
- The provider says the call never ran: you still get `delivery_uncertain` and no refund. A paused worker that holds the claim could still send, so this answer is logged as `mcp_dispatch_status_lookup_not_delivered_unfenced` for your manual reconciliation.
- The lookup is not registered, raises, times out, returns nothing, returns a malformed answer or a non-boolean `delivered`, or returns a result that fails the checks: you still get `delivery_uncertain`.

The gateway never re-sends the call on this path. Your lookup is your own code, so keep it read-only. The lookup must be an `async def` function, an object with an `async def __call__`, or a `functools.partial` of either. A plain function raises `TypeError` at registration, because the timeout cannot interrupt synchronous code. A non-error (`isError` false) result whose `structuredContent` is missing or does not match the tool's `outputSchema` stays `delivery_uncertain`, as on the live path. `isError` results skip the `outputSchema` check. A lookup that returns something other than an awaitable, such as a sync function marked with `inspect.markcoroutinefunction`, also stays `delivery_uncertain`. The gateway never touches an attempt that already has a signed `delivery_uncertain` receipt, so those still need manual reconciliation. For action-permit calls, the query's `forwarded_idempotency_key` is `None` because the native key sent upstream is not recorded, so match on `invocation_id`. The cleanup log field `dispatch_status_resolved` counts the attempts the lookup resolved.
The gateway never re-sends the call on this path. Your lookup is your own code, so keep it read-only. The lookup must be an `async def` function, an object with an `async def __call__`, or a `functools.partial` of either. A plain function raises `TypeError` at registration, because the timeout cannot interrupt synchronous code. A non-error (`isError` false) result whose `structuredContent` is missing or does not match the tool's `outputSchema` stays `delivery_uncertain`, as on the live path. `isError` results skip the `outputSchema` check. A lookup that returns something other than an awaitable, such as a sync function marked with `inspect.markcoroutinefunction`, also stays `delivery_uncertain`. The automatic reconciler never revisits an attempt that already has a signed `delivery_uncertain` receipt. The bootstrap-admin-only operator endpoint can append an evidence-bound resolution; it never changes that receipt or redispatches. The charge stands unless the operator records `confirmed_not_delivered` with evidence and explicitly requests a separate refund. See the [operator resolution runbook](docs/operator-resolution-runbook.md). For action-permit calls, the query's `forwarded_idempotency_key` is `None` because the native key sent upstream is not recorded, so match on `invocation_id`. The cleanup log field `dispatch_status_resolved` counts the attempts the lookup resolved.

---

Expand Down
2 changes: 2 additions & 0 deletions gateway/app/db/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -1013,6 +1013,8 @@ class ReceiptResolutionModel(SQLModel, table=True):
prev_hash: Optional[str] = Field(default=None, max_length=64)
receipt_hash: str = Field(max_length=64, index=True)
outcome: str = Field(max_length=32)
evidence: Optional[str] = Field(default=None, sa_column=Column(Text, nullable=True))
evidence_hash: Optional[str] = Field(default=None, max_length=64)
created_at: datetime = Field(sa_type=NaiveUTCDateTime, default_factory=utc_now)
payload_hash: str = Field(max_length=64)
resolution_hash: str = Field(max_length=64, unique=True, index=True)
Expand Down
1 change: 1 addition & 0 deletions gateway/app/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -838,6 +838,7 @@ def mount_dormant_trust_surfaces(application: FastAPI) -> None:
),
)
app.include_router(api_keys.admin_router)
app.include_router(receipts.admin_router)

# Self-serve dev keys: the handler is triple-gated at runtime (its own
# ENABLE_DEV_KEY_SELF_PROVISION flag answers 404, production-like environments
Expand Down
46 changes: 46 additions & 0 deletions gateway/app/routers/receipts.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
from app.core.wallet_resolution import resolve_wallet_id
from app.schemas.resolutions import (
ResolutionListResponse,
ResolutionRequest,
ResolutionResponse,
resolution_to_response,
)
Expand Down Expand Up @@ -382,3 +383,48 @@ async def verify_receipt(
else:
auth.require_bootstrap_admin()
return ReceiptVerifyResponse(valid=valid, reason=reason, receipt=receipt)


admin_router = APIRouter(prefix="/v1/admin/receipts", tags=["Operator Receipts"])


@admin_router.post("/{receipt_id}/resolution", response_model=ResolutionResponse)
async def resolve_uncertain_receipt(
receipt_id: str,
request: ResolutionRequest,
auth: AuthContext = Depends(get_auth_context),
) -> ResolutionResponse:
from app.services.operator_resolution import resolve_receipt
from app.trust import sha256_hex

error = None
try:
auth.require_bootstrap_admin()
model = await resolve_receipt(receipt_id, request)
return resolution_to_response(model)
except ReceiptResolutionError as exc:
error = exc.reason
raise HTTPException(
status_code=404 if error == "receipt_not_found" else 409, detail=error
) from exc
except HTTPException:
error = "admin_access_denied"
raise
except Exception:
error = "resolution_failed"
raise HTTPException(status_code=503, detail=error) from None
finally:
await record_audit_event(
event="receipt.resolution.attempt",
endpoint=f"/v1/admin/receipts/{receipt_id}/resolution",
auth_source=auth.source,
key_id=auth.key_id,
ok=error is None,
error=error,
metadata={
"receipt_id": receipt_id,
"outcome": request.outcome,
"operator_id": auth.key_id or sha256_hex(auth.raw_key.encode("utf-8")),
"refund_credits": request.refund_credits,
},
)
35 changes: 32 additions & 3 deletions gateway/app/schemas/resolutions.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,20 @@
from __future__ import annotations

import json
from typing import Literal, cast
from typing import Any, Literal, cast

from pydantic import BaseModel
from datetime import datetime
from pydantic import BaseModel, ConfigDict, Field, field_validator

from app.core.config import public_api_origin
from app.db.models import ReceiptResolutionModel
from app.services.signing_keys import RECEIPT_CANONICALIZATION


class ResolutionResponse(BaseModel):
version: Literal["resolution-v0.1"]
version: Literal["resolution-v0.1", "resolution-v0.2"]
evidence: dict[str, Any] | None = None
evidence_hash: str | None = None
issuer: str
resolution_id: str
receipt_id: str
Expand Down Expand Up @@ -47,6 +50,8 @@ def resolution_to_response(model: ReceiptResolutionModel) -> ResolutionResponse:
return ResolutionResponse(
version=signed["version"],
issuer=public_api_origin(),
evidence=json.loads(model.evidence) if model.evidence else None,
evidence_hash=model.evidence_hash,
resolution_id=model.resolution_id,
receipt_id=model.receipt_id,
wallet_id=model.wallet_id,
Expand All @@ -63,3 +68,27 @@ def resolution_to_response(model: ReceiptResolutionModel) -> ResolutionResponse:
signature=model.signature,
kid=model.signature_key_id,
)


class ResolutionEvidence(BaseModel):
model_config = ConfigDict(extra="forbid")
source: str = Field(min_length=1, max_length=256)
upstream_reference_ids: list[str] = Field(default_factory=list, max_length=64)
observed_at: datetime
note: str = Field(min_length=1, max_length=4096)
raw_lookup_answer: dict[str, Any] | None = None

@field_validator("observed_at")
@classmethod
def require_timezone(cls, value: datetime) -> datetime:
if value.tzinfo is None:
raise ValueError("observed_at requires a timezone")
return value


class ResolutionRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
outcome: Literal["confirmed_success", "confirmed_not_delivered"]
evidence: ResolutionEvidence
use_lookup: bool = False
refund_credits: bool = False
12 changes: 11 additions & 1 deletion gateway/app/services/mcp_dispatch_attempts.py
Original file line number Diff line number Diff line change
Expand Up @@ -1883,7 +1883,17 @@ async def mark_debit_refunded(
"dispatch_refund_ledger_conflict"
)
if attempt.state != "returned_error":
raise DispatchAttemptConflictError("dispatch_refund_state_invalid")
from app.services.receipt_resolutions import (
operator_refund_authorized,
)

if (
attempt.state != "delivery_uncertain"
or not await operator_refund_authorized(session, attempt_id)
):
raise DispatchAttemptConflictError(
"dispatch_refund_state_invalid"
)
refund = (
await session.execute(
select(LedgerEntryModel).where(
Expand Down
Loading
Loading