fix(ledger): signature hygiene — reject entity-name tokens, count only signable types - #4123
Merged
Merged
Conversation
The hygiene cron counted every consumable claim with a NULL signaturesBackfilledAt against a threshold of 500, and read 34,320 in production. 34,275 of those are release/new_capability rows the backfill is designed never to stamp, so the count could not fall below the threshold no matter what anyone did. An alert that always fires is the one a reader learns to skim, which is the failure this cron exists to prevent. The change-type list moves from bin/backfillClaimSignatures.ts to src/common/claimSignatures.ts so the counter and the backfill cannot drift apart. Real figure: 45.
|
🍹 The Update (preview) for dailydotdev/api/prod (at ee80dcf) was successful. Resource Changes Name Type Operation
~ vpc-native-check-analytics-report-cron kubernetes:batch/v1:CronJob update
~ vpc-native-update-views-cron kubernetes:batch/v1:CronJob update
~ vpc-native-validate-active-users-cron kubernetes:batch/v1:CronJob update
~ vpc-native-personalized-digest-cron kubernetes:batch/v1:CronJob update
~ vpc-native-channel-digests-cron kubernetes:batch/v1:CronJob update
~ vpc-native-user-profile-updated-sync-cron kubernetes:batch/v1:CronJob update
~ vpc-native-channel-highlights-cron kubernetes:batch/v1:CronJob update
~ vpc-native-refresh-tool-stack-stats-cron kubernetes:batch/v1:CronJob update
~ vpc-native-clean-expired-better-auth-sessions-cron kubernetes:batch/v1:CronJob update
~ vpc-native-post-analytics-clickhouse-cron kubernetes:batch/v1:CronJob update
~ vpc-native-rotate-weekly-quests-cron kubernetes:batch/v1:CronJob update
~ vpc-native-update-tag-materialized-views-cron kubernetes:batch/v1:CronJob update
~ vpc-native-personalized-digest-deployment kubernetes:apps/v1:Deployment update
~ vpc-native-post-lifecycle-state-clickhouse-cron kubernetes:batch/v1:CronJob update
+ vpc-native-api-clickhouse-migration-acf452dd kubernetes:batch/v1:Job create
~ vpc-native-clean-stale-user-transactions-cron kubernetes:batch/v1:CronJob update
~ vpc-native-ws-deployment kubernetes:apps/v1:Deployment update
~ vpc-native-generic-referral-reminder-cron kubernetes:batch/v1:CronJob update
+ vpc-native-api-db-migration-acf452dd kubernetes:batch/v1:Job create
~ vpc-native-update-trending-cron kubernetes:batch/v1:CronJob update
~ vpc-native-interest-scheduled-run-cron kubernetes:batch/v1:CronJob update
~ vpc-native-update-tags-str-cron kubernetes:batch/v1:CronJob update
~ vpc-native-post-analytics-history-day-clickhouse-cron kubernetes:batch/v1:CronJob update
~ vpc-native-update-achievement-rarity-cron kubernetes:batch/v1:CronJob update
~ vpc-native-subscription-anniversary-achievements-cron kubernetes:batch/v1:CronJob update
~ vpc-native-user-posts-analytics-refresh-cron kubernetes:batch/v1:CronJob update
~ vpc-native-rotate-daily-quests-cron kubernetes:batch/v1:CronJob update
~ vpc-native-private-deployment kubernetes:apps/v1:Deployment update
~ vpc-native-update-source-public-threshold-cron kubernetes:batch/v1:CronJob update
~ vpc-native-clean-old-notifications-cron kubernetes:batch/v1:CronJob update
~ vpc-native-materialize-monthly-best-post-archives-cron kubernetes:batch/v1:CronJob update
~ vpc-native-worker-job-deployment kubernetes:apps/v1:Deployment update
~ vpc-native-world-index-cron kubernetes:batch/v1:CronJob update
~ vpc-native-ledger-hygiene-cron kubernetes:batch/v1:CronJob update
~ vpc-native-daily-digest-cron kubernetes:batch/v1:CronJob update
~ vpc-native-temporal-deployment kubernetes:apps/v1:Deployment update
~ vpc-native-bg-deployment kubernetes:apps/v1:Deployment update
~ vpc-native-update-current-streak-cron kubernetes:batch/v1:CronJob update
~ vpc-native-calculate-top-readers-cron kubernetes:batch/v1:CronJob update
~ vpc-native-user-profile-analytics-clickhouse-cron kubernetes:batch/v1:CronJob update
~ vpc-native-hourly-notification-cron kubernetes:batch/v1:CronJob update
... and 20 other changes |
extractClaimSignatures already dropped a token repeating the claim's own entity, on the grounds that it matches every plan mentioning the technology rather than the change. The same argument applies to every other entity the ledger knows, and nothing enforced it: production carries 1,103 distinct signature tokens that are exactly an entity name -- "Claude Code", "Python 3.10", "React 19", "Redis", "Docker" among the most repeated. Tier A is meant to say "your code touches the thing that changed". These say "your plan mentions this technology", which the entity tiers already say version-gated and therefore more precisely, so the rule moves signal to a better tier rather than dropping it. This is the same failure as the generic word list one layer up, where the token is specific but still not a code surface. The live extraction worker never applied the bar's entity half at all; it does now. codeOnlyCanonical names are excluded from the reject set -- those ARE code tokens.
idoshamun
changed the title
#4115 enforced the bar on the two extraction paths, but /claims/update and /candidates/resolve accept affected/superseding straight from a reviewer and applied nothing. A hand-written "name" accuses every codebase on earth exactly as an extracted one does. The generic-word half moves into the shared `signatures` schema, so any route using it inherits the rule. The entity-name half needs the ledger's own names, so the two routes apply it after parsing. Tokens are filtered rather than rejected: the operator's other overrides in the same call are still valid, and a change whose only symbol is generic legitimately carries none.
The first pass rejected any token matching a known entity name, which would have taken 1,738 tokens including `esbuild`, `curl`, `minimatch`, `encoding/json/v2`, `System.Text.Json` and `GPT-5.3-Codex`. Those are entity names AND literal code tokens: they are what a package.json pins, what a file imports, what a request body sets. Dropping them deletes real signal — the same reason the specificity bar deliberately leaves bare lowercase words alone and lets the detector gate the match on the claim's entity being resolved in the same input. A name with a space in it is different in kind: nothing imports "React Server Components" or pins "Swift Package Manager", so a plan holding that string is describing the technology in prose, which is what the entity tiers are for. Both halves are now required. Production figure drops from 1,738 tokens to 434 across 397 claims.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two defects in the signature layer, both found while verifying the ledger is ready for the rot-bench eval.
1. Signature tokens that are just an entity name
extractClaimSignaturesalready dropped a token repeating the claim's own entity, because it matches every plan that mentions the technology rather than the change. The same argument applies to every other entity the ledger knows, and nothing enforced it.Production carries 1,103 distinct signature tokens that are exactly an entity name (1,610 instances across 1,358 claims). The most repeated:
Claude Code,Python 3.10,React 19,Redis,Docker,Node.js 18,GitHub Actions.Tier A is meant to say your code touches the thing that changed. These say your plan mentions this technology — which the entity tiers already say, version-gated and therefore more precisely. The rule moves signal to a better tier rather than dropping it. Same failure as the generic word list one layer up: the token is specific, but still not a code surface. It is also the same shape as the 2026-08-20 incident, where unfiltered generic tokens gave rot-bench's first harness pilot 15 identical Tier A false findings.
src/common/ledgerEntityNames.tsbuilds the prose-name set (canonical unlesscodeOnlyCanonical, plus aliases), cached an hour so it is one query per process.codeOnlyAliasesand code-only canonicals are excluded from the reject set — those ARE code tokens (browser_toolset_20260801).2. The unsigned-claims alert could never stop firing
ledger-hygienecounted every consumable claim with a NULLsignaturesBackfilledAtagainst a threshold of 500, and read 34,320 in production. But 34,275 of those arerelease/new_capabilityrows the backfill is designed never to stamp (signatures make nothing stale for those types — measured at 0/45 and 1/49 fill). The count could not drop below the threshold no matter what anyone did, and an alert that always fires is the one a reader learns to skim.The real number awaiting the pass was 45; they have since been stamped in production (45/45, 26 carrying tokens, 0 failed).
SIGNABLE_CHANGE_TYPESmoves frombin/backfillClaimSignatures.tsintosrc/common/claimSignatures.tsso the counter and the backfill share one list.Follow-up
The 1,610 existing entity-name tokens are cleaned in production separately; this PR stops new ones being written.