fix(replay): persist message drops - #1447
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Critical CL-mode processing and replay-duplication issues, along with other review findings, remain unresolved.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 3
Open (7)
What changed in this PR
Adds durable recovery evidence and live bounded replay/reset operations for the standalone verifier.
Changes:
- Adds recovery persistence, admission auditing, finality handling, and retention.
- Adds resumable recovery CLI workflows and reader/queue coordination.
- Adds observability, dashboards, documentation, migrations, and E2E coverage.
| File | Summary |
|---|---|
verifier/pkg/vtypes/types.go |
Adds source block hash metadata. |
verifier/pkg/sourcereader/service.go |
Integrates recovery and admission handling. |
verifier/pkg/sourcereader/recovery.go |
Implements replay and reader reset workflows. Review: critical (1 vote) replay drops do not enter terminal tracking, allowing duplicate rediscovery; moderate (1 vote) reset audit failures are not surfaced; moderate (1 vote) capacity counts failed/completed rows; moderate (1 vote) registration failures use audit-failure telemetry; nit (2 votes) comment exceeds the three-line limit. |
verifier/pkg/sourcereader/recovery_test.go |
Tests recovery behavior and resets. |
verifier/pkg/sourcereader/recovery_audit.go |
Persists drops and finality incidents. |
verifier/pkg/sourcereader/finality_checker.go |
Captures finality evidence. |
verifier/pkg/sourcereader/finality_checker_test.go |
Tests finality evidence. |
verifier/pkg/sourcereader/admission.go |
Shares admission logic. |
verifier/pkg/recovery/types.go |
Defines recovery records and requests. |
verifier/pkg/recovery/store.go |
Implements event persistence and cleanup. |
verifier/pkg/recovery/store_test.go |
Tests event and transaction storage. |
verifier/pkg/recovery/operations.go |
Implements durable operation state management. |
verifier/pkg/recovery/metrics.go |
Adds recovery metrics. |
verifier/pkg/jobqueue/postgres_queue.go |
Initializes archive metrics. |
verifier/pkg/jobqueue/observability_decorator.go |
Collects archive metrics periodically. |
verifier/pkg/jobqueue/archive.go |
Implements archive inventory classification. Review: moderate (1 vote) read-time classification can cause full archive scans and stale inventory; persist/index the bounded category or add matching indexes. Nit (2 votes): shorten the long code comment. |
verifier/pkg/jobqueue/archive_test.go |
Tests archive inventory behavior. |
verifier/pkg/helpers_test.go |
Updates configured-reader test helpers. |
verifier/pkg/coordinator.go |
Wires recovery into source readers. Review: critical (1 vote) recovery is configured for Chainlink-node integration despite absent migration support, which can stop normal polling; gate it behind standalone schema support. |
verifier/pkg/chainstatus/batcher.go |
Coordinates durable reader resets. |
verifier/pkg/chainstatus/batcher_test.go |
Tests reset coordination. |
verifier/migrations/postgres/00009_source_recovery.sql |
Adds recovery tables. |
protocol/common_types.go |
Adds optional event block hashes. |
integration/pkg/accessors/evm/evm_source_reader.go |
Supplies EVM block hashes. |
docs/runbooks/remediating-stuck-or-dropped-messages.md |
Documents recovery workflows. |
docs/monitoring/verifier-recovery.md |
Documents recovery monitoring. |
docs/monitoring/verifier-recovery-alerts.yaml |
Adds recovery alerting. |
docs/monitoring/verifier-archive-inventory.md |
Documents archive inventory. Review: nit (2 votes) documentation incorrectly claims migration 00009 adds failure_category and covering indexes. |
docs/monitoring/verifier-archive-inventory-alerts.yaml |
Adds archive alerts. |
cmd/verifier/run_ccv_cli.go |
Registers recovery commands. Review: nit (1 vote) missing database URL produces an unhelpful <nil> error instead of actionable guidance. |
cli/recovery/README.md |
Documents recovery commands. |
cli/recovery/commands.go |
Implements recovery CLI commands. |
cli/recovery/commands_test.go |
Tests CLI validation and output. |
cli/jobqueue/README.md |
Updates archive recovery guidance. Review: nit (2 votes) documents failure_category as persisted/exposed although it is only a query-time expression. |
cli/chainstatuses/README.md |
Documents live recovery and fallback behavior. |
changelog/2026-09-11_source_recovery.md |
Describes recovery changes and rollout. Review: nit (1 vote) inaccurately claims migration 00009 persists failure_category and archive indexes; nit (1 vote) references nonexistent 00009_recovery.sql and incorrect archive schema changes. |
changelog/2026-09-10_archive_inventory_and_cli.md |
Documents archive inventory context. |
build/devenv/tests/e2e/verifiercli/recovery.go |
Adds E2E recovery client helpers. |
build/devenv/tests/e2e/verifiercli/client.go |
Adds process identity checks. |
build/devenv/tests/e2e/smoke_recovery_cli_test.go |
Adds recovery and archive E2E tests. |
build/devenv/tests/e2e/smoke_policy_hook_test.go |
Uses live recovery for policy tests. Review: critical (1 vote) failure_category is not populated or emitted, so the assertion fails for every policy rejection. |
build/devenv/tests/e2e/smoke_chain_statuses_cli_test.go |
Tests live disabled-reader reset. |
build/devenv/tests/e2e/smoke_aggregator_message_disablement_rules_test.go |
Tests recovery after disablement drops. |
build/devenv/tests/e2e/recovery_helpers_test.go |
Provides shared recovery helpers. |
build/devenv/tests/e2e/finality_reorg_curse_test.go |
Tests finality and curse recovery scenarios. |
build/devenv/go.sum |
Updates dependency checksums. |
build/devenv/dashboards/verifier_recovery.json |
Adds recovery dashboard. Review: moderate (1 vote) uses the archive-inventory metric for the verifier selector instead of verifier_recovery_collection_success. |
build/devenv/dashboards/verifier_archive_inventory.json |
Adds archive inventory dashboard. |
.github/workflows/test-smoke.yaml |
Runs recovery smoke tests. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
tt-cll
enabled auto-merge
September 22, 2026 22:22
|
Code coverage report:
Files added (in
|
carte7000
approved these changes
Sep 25, 2026
bukata-sa
reviewed
Sep 28, 2026
| BlockTimestamp time.Time | ||
|
|
||
| // BlockHash is optional source evidence supplied by the reader; empty means unavailable. | ||
| BlockHash ByteSlice |
Collaborator
There was a problem hiding this comment.
is that possible to make that required? Maybe for new data only? optional field means branching the logic, backfilling if this field is required for some work
bukata-sa
approved these changes
Oct 2, 2026
This was referenced Oct 2, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Addresses CCIP-13501 (durable drop evidence) and CCIP-13502 (live source-range recovery) for the standalone verifier. Stacks on the archive inventory / CLI work (CCIP-13475/13499/13500), which shipped without a schema change; the durable storage those tickets did not need arrives here.
CCIP-13501 — durable drop & finality-incident evidence (
ccv_recovery_events,verifier/pkg/recovery)ccv recovery eventsqueries by message IDs, owner, source/dest chain, reason, observation window, and source-block range, paginated (before-idcursor) as machine-readable JSON with coverage metadata — an empty page does not imply there was no incident or no affected trafficlast_observed_atand extends retention; history survives restart with explicit retention cleanupCCIP-13502 — live bounded source-range recovery (
ccv_recovery_operations,ccv_recovery_readers,verifier/pkg/sourcereader/recovery.go)ccv recovery replay/list/status/cancel/resumeon the standalone CLI: submit an explicit verifier owner, source chain, and inclusive block range while the process keeps running; an omitted--to-blockis captured as a fixed target at submission (never a moving head); returns a durable operation ID;--request-idis an idempotency keyadmission()shared with live polling); admitted work follows normal verification and policy; existing active jobs ride the queue's uniqueness rules; replay does not revoke earlier attestations or reconcile old archived rowsOne migration (
00009_source_recovery.sql) adds the three tables. Also mergesorigin/mainand reconciles main's span-per-message tracing refactor with this branch's admission/recovery logic insourcereader/service.go.Standalone verifier only. The Chainlink core-node deployment intentionally applies neither the migrations nor the
ccv recoverycommand group; supporting CL mode is out of scope — it adds deployment complexity (conditional migration application, core command wiring, separate schema lifecycles) for no current operator need. Core wiring must stay conditional (see the changelog's Compatibility section); extending this to CL mode, if ever, is a separate effort.Testing
verifier/pkg/recovery(store dedup/retention, operations state machine, metrics),verifier/pkg/sourcereader(admission, recovery chunks, all three drop causes, restart),cli/recovery/cli/jobqueue(commands, postgres store), finality checkergo test ./verifier/pkg/sourcereader/...passes; conflicts resolved insourcereader/service.go,cli/jobqueue/README.md, verifiercli e2e clientChecklist
changelogdirectory)