Parse joint and concurrent resolutions, and keep their preamble recitals - #427
Open
Nitjsefnie wants to merge 5 commits into
Open
Parse joint and concurrent resolutions, and keep their preamble recitals#427Nitjsefnie wants to merge 5 commits into
Nitjsefnie wants to merge 5 commits into
Conversation
Four byte-for-byte govinfo bulk-data files covering the joint and concurrent resolution shapes AgoraDMV#201 is about: an hconres introduced/engrossed pair that both carry a 12-recital <preamble>, plus an hjres and an sjres with none. They live beside tests/fixtures/byte_identity/ rather than in tests/corpus_manifest.toml because the manifest enrolls a bill in the appropriations corpus gates, which these procedural texts carry no amounts for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Joint, concurrent and simple resolutions root at <resolution> and carry
<resolution-body> where a bill carries <legis-body>, so find_bill_body raised
"Could not find bill body in XML" for all of them — a measured 0% parse rate
against 100% for hr/s.
Two defects, not one. Even once the body is found, _extract_metadata read the
bill type off the first letter the old regex captured, which collapsed both
chambers' joint resolutions onto "j" and both concurrent ones onto "n". The
issue's table stops there, but the same root element covers simple resolutions
too: H. RES. / S. RES. parsed happily and were labelled with the *bill* types
"hr"/"s", printing a wrong designator on a real document. So <legis-num> is now
split into a prefix and a number, and the normalized prefix ("H. CON. RES." ->
"HCONRES") is looked up against all eight forms; an unrecognized prefix falls
through to its own normalized form rather than a stray letter. Verified to leave
the bill_type of all 30 distinct <legis-num> spellings in the committed corpus
unchanged, including the unspaced "H.R. 2029".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fixing the body finder alone turns a loud failure into a silent one. <preamble> is a direct child of <resolution>, a sibling of <resolution-body>, so nothing walks it: with only the previous commit's change in place BILLS-119hconres58ih parses to exactly four nodes — masthead, official title, enacting clause, one body section — and all twelve "Whereas" recitals are gone from a report that otherwise looks clean. Roughly 14% of resolutions carry one. extract_front_matter_nodes already exists for exactly this problem (AgoraDMV#48: the form block and enacting clause sit outside the body and were dropped), so the recitals join it rather than getting a new diagnostics channel — the engine has no logging or warnings anywhere to hook into. One node keyed "preamble" for the whole block, not one per <whereas>: those elements carry no id, so a per-recital key could only be positional, and inserting a single recital between versions would re-key every later one and cascade into false diffs. It sorts before the enacting clause because GPO prints form -> recitals -> resolving clause -> body and this function returns nodes in render order. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A reported-stage resolution can carry the committee amendment as PAIRED blocks:
a struck copy and an amended copy, as two <resolution-body> and/or two
<preamble> children. root.find() takes the first, which is the superseded one,
so the previous commits turned a loud failure into a silent wrong answer —
BILLS-119hres137rh.xml parsed clean and rendered the struck recitals and struck
body as though they were the document, capturing 17 of its 34 recitals. The two
variants are not cosmetically different: the committee restyled a recital
("army" -> "Army") and rewrote the body's parenthetical.
Choosing between the variants is an amendment-display feature and the
maintainer's call, so detect the shape and raise with a message naming what was
found. This regresses nothing: all three affected files already raise
"Could not find bill body in XML" on develop. Measured prevalence across every
reported-stage (rh/rs) 119/1 resolution: 3 of 62. The other 59, and every
introduced- and engrossed-stage resolution, still gain the fix.
Both counts are checked independently because the shape varies — of the three,
one has two bodies and no preamble, one two bodies and one preamble, one two of
each. The struck block is not reliably marked either: two of the three leave the
first body's `changed` attribute absent rather than "deleted", which is a second
reason not to guess at picking one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_LEGIS_NUM_TYPES was an identity map — every key mapped to its own lowercase form, which is exactly what the `prefix.lower()` default beside it already produced. Emptying the table entirely left the whole fast gate green at 1413 passed, so it was unreachable in every branch and no test pinned it. The normalization is what actually does the work: strip the <legis-num> prefix to letters and lowercase it, and "H. CON. RES." becomes "hconres" for every form GPO prints. So the table goes, and a new TestLegisNumNormalization pins the mechanism directly — that separators are normalized away (GPO spells the same designator "H. R. 2029" and "H.R. 2029", both in the corpus) and that the result is letters-only and lowercase. Verified to bite by gutting: replacing the normalization with a bare .lower() turns all four cases red. Also corrects the degradation test, whose docstring claimed an unfamiliar form falls through to "the pre-AgoraDMV#201 single-letter behaviour". It does not — it yields "xyres". The test now asserts that value directly instead of asserting absence from _DESIGNATORS, which drops an import of production data from the test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issue
Closes #201
What does this change?
Joint and concurrent resolutions parse instead of raising, and every resolution form is labelled
correctly.
Three parts, in the order they matter:
find_bill_bodynow recognises<resolution-body>, which is what madenormalize_billraise on every resolution._LEGIS_NUM_REcaptures the whole prefix and_extract_metadatanormalises it, soall eight forms map correctly. Before this,
jandnwere chamber-ambiguous rather thanmerely wrong —
hjresandsjreswere indistinguishable, as werehconresandsconres."preamble", ordered beforethe enacting clause, riding the
extract_front_matter_nodesmechanism XML diff: bill front matter (form block + enacting clause) missing from full-bill text and diff #48 added for exactly this("content outside the body was dropped from the full-bill text and the diff").
Your trap-at-pickup note was right, and it went one layer deeper than the issue. Fixing the body
finder alone drops the recitals of the ~14% of resolutions carrying a preamble — that is handled
above. But reported-stage resolutions carry something else: paired committee-amendment variants,
two
<resolution-body>and sometimes two<preamble>children, struck and amended.root.find()takes the first, which is the struck one, so the fix as first written parsed those clean and rendered
the superseded text as the document — turning your loud
ValueErrorinto a quiet wrong answer.They now raise, naming what was found:
Choosing between the variants is a real feature and your call, so this refuses rather than guesses.
Nothing regresses: those files already fail today.
Two things the sweep turned up that shaped the guard. The shape varies — of the three affected
files, one has 2 bodies and 0 preambles, one 2 and 1, one 2 and 2 — so a preamble-only guard would
miss two of three, and both child kinds are counted independently. And the struck block is often
unmarked: in two of three the first
<resolution-body>carries nochangedattribute at all, soa future variant-selection feature cannot key on that attribute either.
The issue's mapping table is incomplete, worth knowing since it will be the reference next time:
simple resolutions (
H. RES./S. RES.) share the<resolution>root, so the body fix makes themparse and then stamps them with the bill designators
H.R./S.on a real document. All eightforms are handled and tested.
How to test
Prevalence was measured, not estimated: all six 119/1 type archives were downloaded from govinfo and
all 2,122 files parsed. Exactly 3 of the 62 reported-stage files carry paired variants
(
hres137rh,hres237rh,sres226rs), and those same 3 are the only ones flagged across everystage.
Fixtures are four real GPO XML files, committed, each byte-identical to what govinfo serves today.
Bite, in both directions:
develop—stock fails them for the wrong reason (the crash) and would prove nothing. With only the 5-line body
fix applied,
hconres 58parses to exactly 4 nodes with 0 of its 12 recitals; the tests fail onabsence.
DID NOT RAISE ValueError— the silent-wrong-answer state..lower()turns 4 red, e.g.'H. CON. RES. 58' -> 'h. con. res.' is not letters-only.The 30 distinct
<legis-num>spellings in the committed corpus are unchanged by the regex swap,including the unspaced
H.R. 2029form; the only three that change are the new resolution fixtures.Gates:
uv run ruff check .clean ·uv run ruff format --check .178 files · fast1417 passed, 15 skipped, 15 xfailedagainst a1403/15/15baseline · browser6 passed· slow564 passed, 7 skippedagainst554/7. Skips unchanged in both tiers, so nothing needed an allowlistentry. Those counts are from this machine and run higher than yours — offered as a delta.
A warning-count delta is deliberately not quoted: three runs of identical code gave 4, 5 and 3, since
xdist dedupes per worker.
Checklist
Closes #...)AI assistance
Generated by Claude Opus 5 (1M context) (brief, implementation, review), Kimi K3 (verification)
Follow-ups / Known Limitations
saw that they get the bill's "Be it enacted…" boilerplate. Out of scope here as the issue scopes it,
but it is the natural next change and I am happy to take it.
normalize_billcalls the bodyfinder first and grep confirms those are the only callers, so a second guard would be unreachable —
which is the same thing you had us remove on fix(fetch): verify the archive opens before caching a download #211.
<form>element exists, inherited from XML diff: bill front matter (form block + enacting clause) missing from full-bill text and diff #48's early return. All149 real 119/1 resolutions sampled have one, and none has a preamble without a form, so the gap is
bounded — but it is XML diff: bill front matter (form block + enacting clause) missing from full-bill text and diff #48's contract, not something to change here.
server/app.py, and the PDF path are all out ofscope and yours to call.