Skip to content

fix: stop pausing paid DIPS deployments and two timing bugs#1240

Open
MoonBoi9001 wants to merge 8 commits into
main-dipsfrom
mb9/fix-dips-allocation-reconciliation
Open

fix: stop pausing paid DIPS deployments and two timing bugs#1240
MoonBoi9001 wants to merge 8 commits into
main-dipsfrom
mb9/fix-dips-allocation-reconciliation

Conversation

@MoonBoi9001

@MoonBoi9001 MoonBoi9001 commented Jun 23, 2026

Copy link
Copy Markdown
Member

TL;DR

Reviewing indexer-agent logs during a Direct Indexer Payments (DIPS) run surfaced three independent bugs: the agent pauses a subgraph it is still being paid to index, plus two timing bugs — it sizes an allocation's lifetime from a value in seconds but reads it as epochs, so the work never winds down, and it reads the current epoch from a clock that can lag the chain, so fresh work is judged stale. This fixes all three. Stacks on the duplicate-log-key PR (#1239).

Motivation

After an indexer accepts a DIPS deal, the agent indexes the subgraph it is paid for and keeps that work running for the life of the deal. Three separate things go wrong while it does that:

  • Pausing a paid deployment. When the agent's in-memory list of what to index briefly drops the entry, the pause path stops a subgraph that still has an active DIPS agreement — a ~75-second gap was seen right after a deal was struck, so the indexer stops earning on work it is contracted to do.
  • Timing bug 1 — allocation lifetime in the wrong unit. The agent decides how long to keep an allocation open from the agreement's collection window — the longest gap allowed between payment collections, which dipper sends in seconds. That lifetime is measured in epochs, but the seconds value was stored without converting, so a one-hour window (3,600 seconds) was read as 3,600 epochs. The allocation then effectively never expires or renews, risking lost rewards once the network treats the work as stale.
  • Timing bug 2 — epoch read from a lagging clock. Each reconciliation pass reads the current epoch from a separate source that can fall behind the chain head, so the agent can judge fresh work as stale and emits logs an operator cannot reconcile against the actual chain.

The fix for each: the pause path leaves a deployment alone while it has an active DIPS agreement; the collection window is converted from seconds to epochs (using the chain's epoch length) before the indexing rule is stored; and the epoch is read once per reconciliation pass and logged with the block it was read at.

MoonBoi9001 and others added 3 commits June 23, 2026 16:10
The agent's root logger stamps every line with name=IndexerAgent, yet some log calls also put
the subgraph's own name under that same name key, so one JSON line carried name twice. Strict
parsers keep only the last value and drop the first; the per-call field is now subgraphName.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KJmMVG736T3xefovirWryK
Add an eslint rule across the agent, common and cli packages banning pino's reserved keys
(name, level, time, pid, hostname, msg, v) as per-call log fields, which silently emit a
duplicate JSON key. It caught two more real collisions (a `name` and a `msg`), fixed here too.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KJmMVG736T3xefovirWryK
The ETH-balance monitor, freshness checkers, and network monitor each re-set `component` on a
child logger that already inherited it, emitting two `component` keys (invalid JSON).
Use a distinct `subComponent` key instead, and drop a duplicate `protocolNetwork`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-project-automation github-project-automation Bot moved this to 🗃️ Inbox in Indexer Jun 23, 2026
@MoonBoi9001 MoonBoi9001 force-pushed the mb9/fix-dips-allocation-reconciliation branch from f7ac28c to 47309f1 Compare June 23, 2026 06:26
MoonBoi9001 and others added 2 commits June 23, 2026 18:32
A DIPS agreement states its collection window in seconds, but allocation lifetime on an indexing
rule is consumed in epochs, so writing the raw seconds (e.g. 86400) made allocations effectively
never expire. Convert seconds to epochs once before the rule is stored.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reconciliation could pause a deployment that still has a live DIPS agreement when a lagging rule
dropped it from the target set, so those stay indexed. It also reads the epoch fresh each pass,
instead of a cached value that lagged the chain, and logs the block the epoch was read at.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@MoonBoi9001 MoonBoi9001 force-pushed the mb9/fix-dips-allocation-reconciliation branch from 47309f1 to b7146e0 Compare June 23, 2026 06:33
MoonBoi9001 and others added 3 commits June 23, 2026 19:00
In manual deployment mode with DIPS enabled, the agent fetched the active DIPS
deployments to build the target set, then reconcileDeployments fetched them again to
do the same. Drop the caller-side fetch so the database and subgraph lookups run once.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BoGTCXdSZjjjtkmyg3MZpP
Several reconcile steps read the DipsManager assuming it exists whenever DIPS is on, but it is
built lazily and can be briefly absent at startup, where an unchecked read crashed the whole
pass. Route them through one helper that hands back the manager only when ready, else skips.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BoGTCXdSZjjjtkmyg3MZpP
Two methods each hard-coded a 12-second block time to turn a duration in seconds into
epochs. Move that derivation into one NetworkMonitor method so the assumption is stated
once, and have both the allocation lifetime and the DIPS collection window read from it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BoGTCXdSZjjjtkmyg3MZpP
Base automatically changed from mb9/fix-duplicate-json-keys-in-agent-logs to main-dips June 23, 2026 21:38
@MoonBoi9001 MoonBoi9001 requested a review from tmigone June 26, 2026 11:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: 🗃️ Inbox

Development

Successfully merging this pull request may close these issues.

1 participant