Skip to content

Parse joint and concurrent resolutions, and keep their preamble recitals - #427

Open
Nitjsefnie wants to merge 5 commits into
AgoraDMV:developfrom
Nitjsefnie-OSC:fix/201-resolution-parsing
Open

Parse joint and concurrent resolutions, and keep their preamble recitals#427
Nitjsefnie wants to merge 5 commits into
AgoraDMV:developfrom
Nitjsefnie-OSC:fix/201-resolution-parsing

Conversation

@Nitjsefnie

Copy link
Copy Markdown
Contributor

Related issue

Closes #201

What does this change?

Joint and concurrent resolutions parse instead of raising, and every resolution form is labelled
correctly.

Three parts, in the order they matter:

  1. The body. find_bill_body now recognises <resolution-body>, which is what made
    normalize_bill raise on every resolution.
  2. The label. _LEGIS_NUM_RE captures the whole prefix and _extract_metadata normalises it, so
    all eight forms map correctly. Before this, j and n were chamber-ambiguous rather than
    merely wrong — hjres and sjres were indistinguishable, as were hconres and sconres.
  3. The preamble. Recitals are captured as one front-matter node keyed "preamble", ordered before
    the enacting clause, riding the extract_front_matter_nodes mechanism XML diff: bill front matter (form block + enacting clause) missing from full-bill text and diff #48 added for exactly this
    ("content outside the body was dropped from the full-bill text and the diff").

Your trap-at-pickup note was right, and it went one layer deeper than the issue. Fixing the body
finder alone drops the recitals of the ~14% of resolutions carrying a preamble — that is handled
above. But reported-stage resolutions carry something else: paired committee-amendment variants,
two <resolution-body> and sometimes two <preamble> children, struck and amended. root.find()
takes the first, which is the struck one, so the fix as first written parsed those clean and rendered
the superseded text as the document — turning your loud ValueError into a quiet wrong answer.
They now raise, naming what was found:

ValueError: Resolution carries paired committee-amendment variants (2 <resolution-body>,
2 <preamble>: resolution-body [deleted, added]; preamble [deleted, added]). Choosing between
the struck and the amended text is not supported.

Choosing between the variants is a real feature and your call, so this refuses rather than guesses.
Nothing regresses: those files already fail today.

Two things the sweep turned up that shaped the guard. The shape varies — of the three affected
files, one has 2 bodies and 0 preambles, one 2 and 1, one 2 and 2 — so a preamble-only guard would
miss two of three, and both child kinds are counted independently. And the struck block is often
unmarked: in two of three the first <resolution-body> carries no changed attribute at all, so
a future variant-selection feature cannot key on that attribute either.

The issue's mapping table is incomplete, worth knowing since it will be the reference next time:
simple resolutions (H. RES. / S. RES.) share the <resolution> root, so the body fix makes them
parse and then stamps them with the bill designators H.R. / S. on a real document. All eight
forms are handled and tested.

How to test

Prevalence was measured, not estimated: all six 119/1 type archives were downloaded from govinfo and
all 2,122 files parsed. Exactly 3 of the 62 reported-stage files carry paired variants
(hres137rh, hres237rh, sres226rs), and those same 3 are the only ones flagged across every
stage.

Fixtures are four real GPO XML files, committed, each byte-identical to what govinfo serves today.

Bite, in both directions:

  • The preamble tests are proven against the body-fix-only state, not against stock develop
    stock fails them for the wrong reason (the crash) and would prove nothing. With only the 5-line body
    fix applied, hconres 58 parses to exactly 4 nodes with 0 of its 12 recitals; the tests fail on
    absence.
  • Gutting the variant guard gives DID NOT RAISE ValueError — the silent-wrong-answer state.
  • Gutting the normalization to a bare .lower() turns 4 red, e.g.
    'H. CON. RES. 58' -> 'h. con. res.' is not letters-only.

The 30 distinct <legis-num> spellings in the committed corpus are unchanged by the regex swap,
including the unspaced H.R. 2029 form; the only three that change are the new resolution fixtures.

Gates: uv run ruff check . clean · uv run ruff format --check . 178 files · fast
1417 passed, 15 skipped, 15 xfailed against a 1403/15/15 baseline · browser 6 passed · slow
564 passed, 7 skipped against 554/7. Skips unchanged in both tiers, so nothing needed an allowlist
entry. Those counts are from this machine and run higher than yours — offered as a delta.

A warning-count delta is deliberately not quoted: three runs of identical code gave 4, 5 and 3, since
xdist dedupes per worker.

Checklist

  • Linked the issue above (Closes #...)
  • Ran the CI gates locally and they pass (see What CI checks)
  • New or changed behavior has tests
  • For a bug fix: the test fails without the fix, and I ran it both ways to check
  • Disclosed AI assistance below, if any

AI assistance

Generated by Claude Opus 5 (1M context) (brief, implementation, review), Kimi K3 (verification)

Follow-ups / Known Limitations

Nitjsefnie and others added 5 commits July 31, 2026 15:21
Four byte-for-byte govinfo bulk-data files covering the joint and concurrent
resolution shapes AgoraDMV#201 is about: an hconres introduced/engrossed pair that both
carry a 12-recital <preamble>, plus an hjres and an sjres with none. They live
beside tests/fixtures/byte_identity/ rather than in tests/corpus_manifest.toml
because the manifest enrolls a bill in the appropriations corpus gates, which
these procedural texts carry no amounts for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Joint, concurrent and simple resolutions root at <resolution> and carry
<resolution-body> where a bill carries <legis-body>, so find_bill_body raised
"Could not find bill body in XML" for all of them — a measured 0% parse rate
against 100% for hr/s.

Two defects, not one. Even once the body is found, _extract_metadata read the
bill type off the first letter the old regex captured, which collapsed both
chambers' joint resolutions onto "j" and both concurrent ones onto "n". The
issue's table stops there, but the same root element covers simple resolutions
too: H. RES. / S. RES. parsed happily and were labelled with the *bill* types
"hr"/"s", printing a wrong designator on a real document. So <legis-num> is now
split into a prefix and a number, and the normalized prefix ("H. CON. RES." ->
"HCONRES") is looked up against all eight forms; an unrecognized prefix falls
through to its own normalized form rather than a stray letter. Verified to leave
the bill_type of all 30 distinct <legis-num> spellings in the committed corpus
unchanged, including the unspaced "H.R. 2029".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fixing the body finder alone turns a loud failure into a silent one. <preamble>
is a direct child of <resolution>, a sibling of <resolution-body>, so nothing
walks it: with only the previous commit's change in place BILLS-119hconres58ih
parses to exactly four nodes — masthead, official title, enacting clause, one
body section — and all twelve "Whereas" recitals are gone from a report that
otherwise looks clean. Roughly 14% of resolutions carry one.

extract_front_matter_nodes already exists for exactly this problem (AgoraDMV#48: the
form block and enacting clause sit outside the body and were dropped), so the
recitals join it rather than getting a new diagnostics channel — the engine has
no logging or warnings anywhere to hook into.

One node keyed "preamble" for the whole block, not one per <whereas>: those
elements carry no id, so a per-recital key could only be positional, and
inserting a single recital between versions would re-key every later one and
cascade into false diffs. It sorts before the enacting clause because GPO prints
form -> recitals -> resolving clause -> body and this function returns nodes in
render order.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A reported-stage resolution can carry the committee amendment as PAIRED blocks:
a struck copy and an amended copy, as two <resolution-body> and/or two
<preamble> children. root.find() takes the first, which is the superseded one,
so the previous commits turned a loud failure into a silent wrong answer —
BILLS-119hres137rh.xml parsed clean and rendered the struck recitals and struck
body as though they were the document, capturing 17 of its 34 recitals. The two
variants are not cosmetically different: the committee restyled a recital
("army" -> "Army") and rewrote the body's parenthetical.

Choosing between the variants is an amendment-display feature and the
maintainer's call, so detect the shape and raise with a message naming what was
found. This regresses nothing: all three affected files already raise
"Could not find bill body in XML" on develop. Measured prevalence across every
reported-stage (rh/rs) 119/1 resolution: 3 of 62. The other 59, and every
introduced- and engrossed-stage resolution, still gain the fix.

Both counts are checked independently because the shape varies — of the three,
one has two bodies and no preamble, one two bodies and one preamble, one two of
each. The struck block is not reliably marked either: two of the three leave the
first body's `changed` attribute absent rather than "deleted", which is a second
reason not to guess at picking one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_LEGIS_NUM_TYPES was an identity map — every key mapped to its own lowercase
form, which is exactly what the `prefix.lower()` default beside it already
produced. Emptying the table entirely left the whole fast gate green at 1413
passed, so it was unreachable in every branch and no test pinned it.

The normalization is what actually does the work: strip the <legis-num> prefix
to letters and lowercase it, and "H. CON. RES." becomes "hconres" for every form
GPO prints. So the table goes, and a new TestLegisNumNormalization pins the
mechanism directly — that separators are normalized away (GPO spells the same
designator "H. R. 2029" and "H.R. 2029", both in the corpus) and that the result
is letters-only and lowercase. Verified to bite by gutting: replacing the
normalization with a bare .lower() turns all four cases red.

Also corrects the degradation test, whose docstring claimed an unfamiliar form
falls through to "the pre-AgoraDMV#201 single-letter behaviour". It does not — it yields
"xyres". The test now asserts that value directly instead of asserting absence
from _DESIGNATORS, which drops an import of production data from the test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parser: handle joint/concurrent resolutions (hjres, sjres, hconres, sconres)

1 participant