P4: the business_cycle snapshot set — three World Bank tables and a FRED composite - #114
Merged
Conversation
…RED composite Stage 1 of re-enabling business_cycle in lecture-wasm (lecture-wasm#70): every live read the lecture makes now has a snapshot here, with intro keeping its live calls as the lesson. - builders/_fred.py: the shared Fred class (fredgraph.csv over urllib, DATE index, `.` -> NaN, server-side frequency change). No pandas_datareader in a builder — the decision on #26. - builders/business_cycle_fred.py -> lectures/us_business_cycle_monthly.csv: UNRATE, USREC, UMCSENT, CPILFESL, INDPRO and the 1929-42 NBER unemployment series on one monthly grid from 1919-01. validate() places every null exactly (series starts, UMCSENT's sparse pre-1978 years, the 2025-10 shutdown hole in UNRATE and CPILFESL) and bounds revisions per series. cadence: monthly — the first. - builders/business_cycle.py now writes three tables from one WDI fetch: business_cycle_data.csv grown from five economies to the nine the lecture plots (free while it has no consumer), unemployment_rate_annual.csv (USA FRA GBR JPN) and private_credit_to_gdp.csv (GBR). Nulls are placed, not counted: only before an economy's first observation or in the newest two years. All three validate before any writes. --summary-json emits a list. - scripts/snapshots.py accepts a list summary and compares a date-shaped date_range.end as text (the stamp's read-back guard caught the mismatch). - Manifests for the three new files (names provisional pending #113), business_cycle_data.csv.yml updated for nine economies, all four stamped by snapshots.py (retrieved, sha256, verified, date_range.end); migration.yml records (landed, P4); CATALOG.md regenerated; PLAN P4 box reframed; builders/README and AGENTS record the per-source rule. UMCSENT's licence is recorded from FRED's own classification ("copyrighted: citation required") as permitted-with-attribution, the condition this repo's policy meets. Gates: check_consumed_files 44/0; strict audit adds no problems; all four snapshots list as verified and not due. See #26, #113, #14; QuantEcon/lecture-wasm#70. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds the remaining business_cycle snapshot set needed to re-enable the lecture in lecture-wasm by introducing three new published datasets (two WDI annual tables and one composite monthly FRED table), plus the corresponding builders and a small snapshots.py enhancement to support multi-file builder summaries.
Changes:
- Add three new datasets + manifests:
unemployment_rate_annual.csv,private_credit_to_gdp.csv, andus_business_cycle_monthly.csv(dynamic snapshots; annual/monthly cadences). - Extend
builders/business_cycle.pyto build/validate/write a 3-file WDI set; addbuilders/_fred.pyandbuilders/business_cycle_fred.pyfor the composite FRED monthly dataset. - Update
scripts/snapshots.py,PLAN.md,migration.yml, and regenerateCATALOG.md/ builders docs to reflect the new snapshot set and stamping behavior.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/snapshots.py | Supports multi-file builder summaries during stamp/pr-body generation; adjusts date-shaped end comparison. |
| PLAN.md | Reframes P4 scope/status to the broader business_cycle snapshot set. |
| migration.yml | Registers the three newly landed datasets under pilot P4. |
| lectures/us_business_cycle_monthly.csv.yml | Manifest for new monthly composite FRED snapshot (provisional filename, monthly cadence). |
| lectures/us_business_cycle_monthly.csv | New monthly composite dataset (1919-01..2026-07 grid). |
| lectures/unemployment_rate_annual.csv.yml | Manifest for new WDI unemployment annual snapshot (4 economies). |
| lectures/unemployment_rate_annual.csv | New annual unemployment dataset (wide YR#### columns). |
| lectures/private_credit_to_gdp.csv.yml | Manifest for new WDI private credit-to-GDP annual snapshot (GBR). |
| lectures/private_credit_to_gdp.csv | New annual private credit dataset (wide YR#### columns). |
| lectures/business_cycle_data.csv.yml | Updates GDP-growth snapshot description/shape to 9 economies; stamps integrity/date_range updates. |
| lectures/business_cycle_data.csv | Expands GDP-growth data to include additional economies. |
| CATALOG.md | Regenerates catalog counts and adds the three new datasets. |
| builders/README.md | Documents the new shared _fred.py and the expanded business_cycle builder set. |
| builders/business_cycle.py | Refactors builder to produce/validate/write a 3-file WDI set with per-file summaries. |
| builders/business_cycle_fred.py | New builder for the composite monthly FRED snapshot, using _fred.py. |
| builders/_fred.py | New shared FRED fetch helper used by FRED builders. |
| AGENTS.md | Records the “one builder per source + composite + shared fetch lib” decision and references _fred.py. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…ast before validate snapshots.py: the dataset-mismatch guard moves into _load_summary(), so pr-body refuses a foreign or missing summary exactly as stamp does instead of raising KeyError. _fred.py: the query string goes through urllib.parse.urlencode. business_cycle_fred.py: pre_process() no longer casts USREC — a `.` from FRED would have raised ValueError with exit 1 (a retry) instead of a ValidationError with exit 2 (a human); validate() now asserts USREC is complete and 0/1, and run() casts after validation. Output bytes unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stage 1 of re-enabling
business_cycleinlecture-wasm(QuantEcon/lecture-wasm#70): after this PR, every live read that lecture makes has a snapshot here.lecture-python-introkeeps its live calls — the API is the lesson there — so this changes nothing a reader sees today; it creates the files the wasm edition will read.Design, as decided today. One builder per source, a composite file where a lecture reads several series together, and a shared fetch library —
builders/_fred.py, theFredclass overfredgraph.csv, so a FRED builder's fetch is one line and itsvalidate()is the only thing worth reading. Nopandas_datareaderin a builder (the recommendation on #26, now acted on). Filenames of the three new files are provisional per the naming-policy discussion (#113) and free to change whileconsumersis empty;business_cycle_data.csvkeeps its name.What lands
business_cycle_data.csvbusiness_cycle.pyunemployment_rate_annual.csv(new)business_cycle.pySL.UEM.TOTL.NE.ZS, USA FRA GBR JPNprivate_credit_to_gdp.csv(new)business_cycle.pyFS.AST.PRVT.GD.ZS, GBRus_business_cycle_monthly.csv(new)business_cycle_fred.pyon_fred.pyValidation is about where nulls are, not how many. The World Bank tables allow a null only before an economy's first observation (UK unemployment starts 1971, France 1970) or in the newest two years (private credit's 2025 is unpublished) — never inside a series. The FRED composite declares every structural null exactly: series starts, UMCSENT's sparse pre-1978 sampling, the historical series' 1942-06 end, and the 2025-10 hole in UNRATE and CPILFESL (the shutdown month BLS never published) — so a second such month fails the refresh for a human rather than shipping. Revisions are bounded per series (CPILFESL and INDPRO are revised; USREC never). A builder that writes a set validates all three before writing any.
Licence. World Bank: CC BY-4.0. FRED: five series are federal/NBER; UMCSENT is the one third-party series, and FRED's page classifies it "copyrighted: citation required" — permitted with attribution, which is what this repo's cache-with-attribution policy provides, so
redistribution: permittedwith the reasoning in the note. Nothing new for #35.Also in this PR.
scripts/snapshots.pyaccepts a list summary from a multi-file builder and compares a date-shapeddate_range.endas text — the stamp's read-back guard (from the Copilot review on #110) caught exactly that mismatch on the first try, refused to write, and this is the fix. All four manifests were stamped bysnapshots.pyrather than by hand.Gates.
check_consumed_files: 44 manifests, 0 errors. Strict audit against local clones: no new problems (several old ones cleared because the clones were fast-forwarded).snapshots.py: four snapshots listed, allverified, none due; the weekly canary will exercise both builders from Monday.Next (QuantEcon/lecture-wasm#70): the wasm lecture edit reading these four URLs, then the flip here —
consumerswithon_refresh: rebuild,migration.yml → repointed.See #26, #113, #14.
🤖 Generated with Claude Code