feat(ledger): extract claims from a lane that never becomes posts - #4140
Merged
Conversation
The claim ledger only ever saw content that reached the feed, which left out the sources that state changes first — vendor changelogs, release notes, security advisories — because they make poor feed content. Adds a second lane over `yggdrasil.v1.ledger-document-published`: yggdrasil fetches and cleans the document, daily-api owns the record and the extraction, because the specificity bar and the statement dedupe are DB-backed here and must not be duplicated in Go. - `ledger_document` holds the record. No source table: name and class are denormalised onto the document, so there is no registry to keep in sync with yggdrasil. `extractedAt` is a timestamp rather than an existence check, so a document that yields nothing is not re-extracted, and re-billed, on every redelivery. - `claim_candidate` and `claim_evidence` gain a nullable `documentId`, `claim_candidate.postId` becomes nullable, and a CHECK holds the candidate to exactly one lane. No post id is synthesised for a document. - `canonicalDocumentUrl` is added beside `normalizeEvidenceUrl` rather than replacing it: url is the cross-lane identity key and a feed hands us the same document wearing `?utm_source=rss`, but the weaker rule's output is already persisted in `claim_evidence.url`. - The extraction core moves to `src/common/claimExtraction.ts`. The lanes differ only in how they resolve content and which key they stamp; the rules that decide what the ledger is worth now exist once. - The lane has no `change_signal` gate. The feed lane can afford one because enrichment had already paid for the triage; this lane skips enrichment entirely. - Whichever lane arrives second at a url the other already extracted stands down, so one document is not paid for twice.
|
🍹 The Update (preview) for dailydotdev/api/prod (at cd06370) was successful. Resource Changes Name Type Operation
~ vpc-native-update-trending-cron kubernetes:batch/v1:CronJob update
~ vpc-native-update-achievement-rarity-cron kubernetes:batch/v1:CronJob update
~ vpc-native-private-deployment kubernetes:apps/v1:Deployment update
~ vpc-native-hourly-notification-cron kubernetes:batch/v1:CronJob update
~ vpc-native-user-profile-updated-sync-cron kubernetes:batch/v1:CronJob update
~ vpc-native-rotate-weekly-quests-cron kubernetes:batch/v1:CronJob update
~ vpc-native-update-source-public-threshold-cron kubernetes:batch/v1:CronJob update
~ vpc-native-update-tags-str-cron kubernetes:batch/v1:CronJob update
~ vpc-native-clean-zombie-images-cron kubernetes:batch/v1:CronJob update
~ vpc-native-channel-digests-cron kubernetes:batch/v1:CronJob update
~ vpc-native-clean-old-notifications-cron kubernetes:batch/v1:CronJob update
~ vpc-native-sync-subscription-with-cio-cron kubernetes:batch/v1:CronJob update
~ vpc-native-clean-zombie-opportunities-cron kubernetes:batch/v1:CronJob update
~ vpc-native-refresh-tool-stack-stats-cron kubernetes:batch/v1:CronJob update
~ vpc-native-materialize-monthly-best-post-archives-cron kubernetes:batch/v1:CronJob update
~ vpc-native-user-profile-analytics-clickhouse-cron kubernetes:batch/v1:CronJob update
~ vpc-native-user-world-clickhouse-cron kubernetes:batch/v1:CronJob update
~ vpc-native-expire-super-agent-trial-cron kubernetes:batch/v1:CronJob update
~ vpc-native-materialize-yearly-best-post-archives-cron kubernetes:batch/v1:CronJob update
~ vpc-native-interest-scheduled-run-cron kubernetes:batch/v1:CronJob update
~ vpc-native-personalized-digest-cron kubernetes:batch/v1:CronJob update
~ vpc-native-clean-expired-better-auth-sessions-cron kubernetes:batch/v1:CronJob update
~ vpc-native-calculate-top-readers-cron kubernetes:batch/v1:CronJob update
~ vpc-native-clean-channel-highlights-cron kubernetes:batch/v1:CronJob update
- vpc-native-api-db-migration-892ca99d kubernetes:batch/v1:Job delete
~ vpc-native-ledger-corroboration-cron kubernetes:batch/v1:CronJob update
~ vpc-native-deployment kubernetes:apps/v1:Deployment update
~ vpc-native-user-posts-analytics-refresh-cron kubernetes:batch/v1:CronJob update
~ vpc-native-ledger-hygiene-cron kubernetes:batch/v1:CronJob update
+ api-sub-api.ledger-document-published-extract-claims gcp:pubsub/subscription:Subscription create
~ vpc-native-ws-deployment kubernetes:apps/v1:Deployment update
~ vpc-native-post-analytics-achievements-cron kubernetes:batch/v1:CronJob update
~ vpc-native-channel-highlights-cron kubernetes:batch/v1:CronJob update
~ vpc-native-update-views-cron kubernetes:batch/v1:CronJob update
~ vpc-native-worker-job-deployment kubernetes:apps/v1:Deployment update
~ vpc-native-daily-digest-cron kubernetes:batch/v1:CronJob update
~ vpc-native-squad-posts-analytics-refresh-cron kubernetes:batch/v1:CronJob update
~ vpc-native-update-highlighted-views-cron kubernetes:batch/v1:CronJob update
~ vpc-native-update-current-streak-cron kubernetes:batch/v1:CronJob update
~ vpc-native-bg-deployment kubernetes:apps/v1:Deployment update
~ vpc-native-rotate-daily-quests-cron kubernetes:batch/v1:CronJob update
... and 22 other changes |
…ne check Review feedback on #4140. The document row no longer mints an id of its own: `LedgerDocument.id` IS the `document_id` yggdrasil sends, and the upsert conflicts on it. That forces the url index to become plain — with the id as the upsert target, a re-poll of the same url under a different document id would stop being an upsert and become a constraint violation raised inside the worker. Uniqueness moves to the id; the url is still stored canonicalised and still indexed, because the cross-lane collision check only asks whether any extracted document holds it, which needs no uniqueness. `CHK_claim_candidate_one_source` goes with it. The extraction target is a union, so neither write path can express a candidate that claims both lanes or neither — the constraint was defence in depth, not load bearing. Nullable `postId`, both partial unique indexes and the foreign keys stay. The migration is amended rather than stacked on: it has not run anywhere but a local database, and it should read as though the table was always shaped this way.
…-api # Conflicts: # src/common/claimLedger.ts # src/common/ledgerHygiene.ts # src/routes/private/ledger.ts # src/workers/extractClaims.ts
Comments only, no behaviour change. Cut the restatements, the incident history and row counts, the playbook citations and the justifications for roads not taken, and compressed what was left to a sentence or two. What stays is the handful a reader would otherwise reintroduce a bug without: why `publishedDate` goes to bragi empty rather than dated today, why `canonicalDocumentUrl` sits beside `normalizeEvidenceUrl` instead of replacing it, why the race guard reads the filed statements twice, and why the document lane withholds its crawl date.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The claim ledger only ever saw content that reached the feed. That leaves out the sources that state a change first — vendor changelogs, release notes, security advisories — because they make poor feed content.
This adds the daily-api half of a second lane over
yggdrasil.v1.ledger-document-published(schema #193). Yggdrasil fetches and cleans the document; daily-api owns the record and the extraction, because the specificity bar and the statement dedupe are DB-backed here. Bragi needs no change —ExtractClaimsRequest.post_idis correlation-only.What's here
ledger_documentholds the record, keyed by thedocument_idyggdrasil sends — the ledger mints no id of its own. Deliberately no source table: name and class are denormalised onto the document, so there is no registry to keep in sync with yggdrasil.extractedAtis a timestamp rather than "does any candidate exist", so a document that yields zero claims is not re-extracted, and re-billed, on every redelivery — which is what the post lane does today.contentHashis what reopens a rolling release-notes page that keeps one document while gaining entries.claim_candidateandclaim_evidencegain a nullabledocumentId, andclaim_candidate.postIdbecomes nullable. No post id is synthesised for a document; the extraction target is a union, so neither write path can express a candidate claiming both lanes or neither.canonicalDocumentUrlis added besidenormalizeEvidenceUrl, not in place of it: url is the cross-lane identity key and a feed hands us the same document wearing?utm_source=rss,www.or plain http — but the weaker rule's output is already persisted inclaim_evidence.url.src/common/claimExtraction.ts. The two lanes differ only in how they resolve content and which key they stamp, so the rules that decide what the ledger is worth exist once.change_signalgate on the new lane. The feed lane can afford one only because enrichment had already paid for the triage; this lane skips enrichment entirely. Intended, not an oversight.Worth a second look
documentId— so if yggdrasil ever re-publishes a document under a fresh id, the same entries would be extracted twice with nothing to dedupe them. Nothing here guards against that, because the right answer depends on yggdrasil's id semantics: if ids are stable per document, this cannot happen; if they are per fetch, thecontentHashreopen never fires either and the lane needs an intra-lane url guard instead.IDX_claim_candidate_postId_statement_unique. A null never equals a null in a btree, so the existing index already binds only rows that have a postId — recreating it would mean reproducing the cutover timestamp baked into its predicate and rebuilding a large index for no behavioural change.down()deletes rows with no postId before restoring NOT NULL; there is no post for them to fall back to.Notes
.infra/common.ts; the topic itself still has to be created in the streams repo.@dailydotdev/schemais bumped 0.3.17 → 0.3.18 for the generatedLedgerDocumentPublishedMessage. Enum numbering differs from bragi'sContentFormat(ledger XML is 1, bragi's is 0), so the mapping is written out rather than cast.pnpm run buildandpnpm run lintpass; migration applies, rolls back and reapplies cleanly, and generating against it shows no drift.