Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions changelog.d/uk-childcare-bus-targets-834-789.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Add five UK childcare calibration targets and two active England bus-finance targets, with six signed bus exclusions (#834, #789).
The activated-reference receipt of monetary bindings now carries the reference's `value_operands` and `expected_member_count`, so a multi-member declaration stays byte-identical through activation.
1 change: 1 addition & 0 deletions changelog.d/uk-childcare-fitter-834.changed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Rewrite the UK childcare take-up fitter: targets compiled from the calibration contract and the pinned Chronicle feed, four rates only with the hours distribution frozen by adjudication, and an expected-count objective (two engine runs with the flags forced on; each row's expectation is linear in its own rate and bilinear with the extended rate through the engine's mutual exclusions) that is exact for the spine's persisted draws, smooth for the optimizer, and reports each row's ceiling at a take-up of one in its provenance receipt (#834).
4 changes: 4 additions & 0 deletions changelog.d/uk-chronicle-repin-834.changed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Re-pin the UK national and local target surfaces to the Chronicle `6fb700e`
consumer artifact, tighten Universal Credit and CGT selectors, align Universal
Credit rows on 2025 calendar-year averages, and activate the available PIPR
local rent rows on the same temporal basis.
1 change: 1 addition & 0 deletions changelog.d/uk-engine-floor-834.changed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Floor the uk extra at policyengine-uk>=2.93 and lock 2.94.0, the first release that defines `tax_free_childcare_spend_routed_share` and applies the gross-side Tax-Free Childcare rate (#834).
1 change: 1 addition & 0 deletions changelog.d/uk-tfc-routed-share-834.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add the HMRC-derived Tax-Free Childcare routed-spend share to the UK person spine and release contract (#834).
26 changes: 26 additions & 0 deletions docs/uk-chronicle-feed-repin.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Re-pin the UK Chronicle consumer feed

The national and local target surfaces must always use one reviewed Chronicle
consumer artifact. Rebuild the UK bundle and consumer artifact in
`PolicyEngine/arch-data`, then copy the resulting `consumer_facts.jsonl` and
`manifest.json` into `.codex-work/consumer_facts_uk.jsonl` and
`.codex-work/consumer_facts_uk_manifest.json`. Do not commit either
file.

Verify both SHA-256 digests and the manifest's `facts_sha256`, row count, and
schema version. Update `_LEDGER_FACT_FEED_PIN` in
`uk_runtime/local_target_census.py`, the local validation-level pin, and their
tests together. Regenerate the local census with
`uv run --no-sync python tools/census_uk_local_targets.py`.

Regenerate the national and local reference surfaces from the same fact file
with `tools/generate_uk_target_references.py` and
`tools/generate_uk_local_target_references.py`, then rebuild the signed compile
parity receipts with
`tools/build_uk_ledger_compile_parity_signed_differences.py`. The hermetic
regeneration test accepts either the default `.codex-work` files or a
`CHRONICLE_UK_FACTS` override and skips only when neither is present.

The calibration runner refuses a feed whose facts digest differs from the
committed pin. `--allow-unpinned-feed` is an explicit diagnostic override and
is recorded in the run manifest; it is not a re-pin procedure.
463 changes: 463 additions & 0 deletions experiments/834-childcare-tfc-receipts.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion packages/microcosm-build/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ us = [
# The UK extra adds the rules engine for local metric generation from a
# Microcosm UK H5. Target tables remain explicit inputs, and the base package
# still does not import policyengine-uk at import time.
uk = ["policyengine-uk>=2.92", "h5py>=3", "tables>=3"]
uk = ["policyengine-uk>=2.93", "h5py>=3", "tables>=3"]

[project.scripts]
microcosm-export-us-l0-refit-h5 = "microcosm.build.us_runtime.l0_refit_export:main"
Expand Down
153 changes: 145 additions & 8 deletions packages/microcosm-build/src/microcosm/build/ledger_targets.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,12 +23,21 @@
ALLOWED_ASSERTION_POLICIES = frozenset(("observed_only", "allow_source_projection"))
ALLOWED_PERIOD_MATCH_POLICIES = frozenset(("latest_not_after", "exact"))
ALLOWED_VALUE_OPERATIONS = frozenset(
("identity", "sum", "calendar_year_average", "latest_plateau", "count_x_mean")
(
"identity",
"sum",
"difference",
"calendar_year_average",
"latest_plateau",
"count_x_mean",
)
)
MULTI_FACT_VALUE_OPERATIONS = frozenset(
("sum", "calendar_year_average", "latest_plateau", "count_x_mean")
("sum", "difference", "calendar_year_average", "latest_plateau", "count_x_mean")
)
EXACT_PERIOD_VALUE_OPERATIONS = frozenset(
("identity", "sum", "difference", "count_x_mean")
)
EXACT_PERIOD_VALUE_OPERATIONS = frozenset(("identity", "sum", "count_x_mean"))
DEFAULT_HIERARCHY_MATCH_SPEC_FIELDS = ("entity", "period", "family", "filter")


Expand Down Expand Up @@ -71,6 +80,8 @@ class LedgerTargetReference:
ledger_source_record_id: str = ""
ledger_selector: Mapping[str, object] = field(default_factory=dict)
value_operation: str = "identity"
value_operands: tuple[Mapping[str, object], ...] = ()
expected_member_count: int | None = None
entity: str = ""
measure: str | None = None
filter: str | None = None
Expand Down Expand Up @@ -111,6 +122,22 @@ def __post_init__(self) -> None:
f"LedgerTargetReference {self.name!r}: ledger_selector must be a "
f"mapping, got {type(self.ledger_selector).__name__}."
)
if self.expected_member_count is not None and (
isinstance(self.expected_member_count, bool)
or not isinstance(self.expected_member_count, int)
or self.expected_member_count <= 0
):
raise ValueError(
f"LedgerTargetReference {self.name!r}: expected_member_count "
"must be a positive integer."
)
if self.value_operation == "difference":
roles = [str(operand.get("role")) for operand in self.value_operands]
if roles != ["minuend", "subtrahend"]:
raise ValueError(
f"LedgerTargetReference {self.name!r}: difference requires "
"exactly ordered minuend/subtrahend operands."
)
if self.assertion_policy not in ALLOWED_ASSERTION_POLICIES:
raise ValueError(
f"LedgerTargetReference {self.name!r}: unsupported "
Expand Down Expand Up @@ -590,6 +617,13 @@ def target_spec_from_ledger_reference(
numeric_value = numeric_values[0] * numeric_values[1]
elif reference.value_operation == "sum":
numeric_value = sum(numeric_values)
elif reference.value_operation == "difference":
numeric_value = numeric_values[0] - numeric_values[1]
if not math.isfinite(numeric_value) or numeric_value < 0:
raise ValueError(
f"Ledger target reference {reference.name!r}: difference "
f"produced invalid value {numeric_value!r}."
)
else:
numeric_value = numeric_values[0]
representative_fact = _value_representative_fact(
Expand Down Expand Up @@ -948,6 +982,8 @@ def _resolve_reference_fact(
eligible_matches = _eligible_selector_matches(reference, matches)
if reference.value_operation == "sum" and eligible_matches:
return _resolve_sum_reference_facts(reference, eligible_matches)
if reference.value_operation == "difference" and eligible_matches:
return _resolve_difference_reference_facts(reference, eligible_matches)
if reference.value_operation == "calendar_year_average" and eligible_matches:
return _resolve_calendar_year_average_reference_facts(
reference, eligible_matches
Expand Down Expand Up @@ -1020,6 +1056,15 @@ def _resolve_sum_reference_facts(
if period_key == latest_period
for fact in facts
]
if (
reference.expected_member_count is not None
and len(latest_matches) != reference.expected_member_count
):
raise ValueError(
f"Ledger target reference {reference.name!r}: value_operation=sum "
f"expected {reference.expected_member_count} members at the latest "
f"period but resolved {len(latest_matches)}; a declared member is missing."
)
return tuple(
sorted(
latest_matches,
Expand All @@ -1028,6 +1073,54 @@ def _resolve_sum_reference_facts(
)


def _resolve_difference_reference_facts(
reference: LedgerTargetReference,
eligible_matches: list[object],
) -> tuple[object, ...]:
resolved: list[object] = []
for operand in reference.value_operands:
dimensions = operand.get("dimension_values")
if not isinstance(dimensions, Mapping):
raise ValueError(
f"Ledger target reference {reference.name!r}: difference operand "
"requires dimension_values."
)
shared_dimensions = reference.ledger_selector.get("dimension_values")
selector = {
**dict(reference.ledger_selector),
**{
str(key): value
for key, value in operand.items()
if key not in {"role", "dimension_values"}
},
"dimension_values": {
**(
dict(shared_dimensions)
if isinstance(shared_dimensions, Mapping)
else {}
),
**dict(dimensions),
},
}
matches = [
fact for fact in eligible_matches if _fact_matches_selector(fact, selector)
]
match = _latest_period_selector_match(reference, matches)
if match is None:
raise ValueError(
f"Ledger target reference {reference.name!r}: difference operand "
f"{operand.get('role')!r} did not resolve exactly once."
)
resolved.append(match)
periods = {_period_key(fact) for fact in resolved}
if len(periods) != 1:
raise ValueError(
f"Ledger target reference {reference.name!r}: difference operands "
"must resolve at the same latest period."
)
return tuple(resolved)


def _resolve_calendar_year_average_reference_facts(
reference: LedgerTargetReference,
eligible_matches: list[object],
Expand Down Expand Up @@ -1292,6 +1385,13 @@ def _validate_resolved_reference_fact(
f"vintage {vintage_pin!r}, but resolved fact has vintage "
f"{_at(fact, 'geography', 'vintage')!r}."
)
entity_pin = reference.ledger_selector.get("entity_name")
if entity_pin is not None and _str_at(fact, "entity", "name") != entity_pin:
raise ValueError(
f"Ledger target reference {reference.name!r} requires entity_name "
f"{entity_pin!r}, but resolved fact has entity "
f"{_at(fact, 'entity', 'name')!r}."
)


def _validate_reference_period(fact: object, reference: LedgerTargetReference) -> None:
Expand Down Expand Up @@ -1333,7 +1433,9 @@ def _latest_period_selector_match(
reference: LedgerTargetReference,
matches: list[object],
) -> object | None:
if len(matches) < 2:
if len(matches) == 1:
return matches[0]
if not matches:
return None
semantic_keys = {_selector_period_invariant_key(fact) for fact in matches}
if len(semantic_keys) != 1:
Expand Down Expand Up @@ -1409,6 +1511,8 @@ def _is_period_fragment(value: str) -> bool:
normalized = value.lower().replace("-", "_")
if _is_period_token(normalized):
return True
if normalized.startswith("year") and _is_period_token(normalized[4:]):
return True
month_names = (
"jan",
"january",
Expand Down Expand Up @@ -1449,15 +1553,39 @@ def _is_period_token(value: str) -> bool:
return bool(period_key[0]) and 1000 <= period_key[1] // 100 <= 9999


def _comparable_period_value(fact: object) -> object:
"""The period value the surface compares a fact on.

Every fiscal-year label on the surface is read as the opening year
(OBR's and HMRC's ``2025`` is FY2025-26). DfT labels its reporting year
by the March end year (``2025`` is YE March 2025, FY2024-25) and says so
in the fact's own coverage dates, so a fiscal-year fact that carries
``period_coverage.start_date`` is keyed on that start year. Facts without
coverage keep their label; the publisher's label is recorded beside the
comparable value (``ledger_fact_period_label``) whenever they differ.
"""

value = _at(fact, "period", "value")
if _str_at(fact, "period", "type") != "fiscal_year":
return value
start = _str_at(fact, "period_coverage", "start_date")
if len(start) < 4 or not start[:4].isdigit():
return value
label = str(value).strip()
if not label.isdigit():
return value
return int(start[:4])


def _period_key(fact: object) -> tuple[int, int, str]:
return _period_key_from_value(_at(fact, "period", "value"))
return _period_key_from_value(_comparable_period_value(fact))


def _exact_period_matches(fact: object, reference: LedgerTargetReference) -> bool:
"""Match exact periods by semantic value while retaining period-kind pins."""

expected_value = reference.period
actual_value = _at(fact, "period", "value")
actual_value = _comparable_period_value(fact)
actual_type = _str_at(fact, "period", "type")
selector_type = str(reference.ledger_selector.get("period_type", ""))
expected_type_hint = period_type_hint(expected_value)
Expand All @@ -1482,7 +1610,7 @@ def _reference_period_partition_key(
) -> tuple[int, int, str]:
period_key = (
_normalize_period_value(
_at(fact, "period", "value"),
_comparable_period_value(fact),
declared_type=_str_at(fact, "period", "type"),
)[0]
if reference.period_match_policy == "exact"
Expand Down Expand Up @@ -1641,6 +1769,8 @@ def _reference_metadata(reference: LedgerTargetReference) -> dict[str, str]:
metadata["ledger_value_operation"] = reference.value_operation
metadata["ledger_assertion_policy"] = reference.assertion_policy
metadata["ledger_period_match_policy"] = reference.period_match_policy
if reference.value_operation == "difference":
metadata["ledger_value_formula"] = "minuend - subtrahend"
for key, value in sorted(reference.ledger_selector.items()):
if isinstance(value, Mapping):
continue
Expand Down Expand Up @@ -1982,7 +2112,14 @@ def _ledger_metadata(fact: object, *, fact_key: str) -> dict[str, str]:
"ledger_legal_vintage": _str_at(fact, "measure", "legal_vintage")
or _str_at(fact, "concept_alignment", "legal_vintage"),
"ledger_period_type": _str_at(fact, "period", "type"),
"ledger_fact_period": _str_at(fact, "period", "value"),
"ledger_fact_period": str(_comparable_period_value(fact)),
# Stamped only when the publisher's label differs from the comparable
# period (DfT's March-end fiscal labels); absent otherwise.
"ledger_fact_period_label": (
_str_at(fact, "period", "value")
if str(_comparable_period_value(fact)) != _str_at(fact, "period", "value")
else ""
),
# Recorded only when the fact asserts it; legacy rows that omit the
# field are not stamped (readers treat absence as
# observation-by-default, same as the artifact loader).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@
"assign_binary_from_rate",
"assign_binary_with_anchored_residual",
"assign_clipped_normal",
"assign_period_constant",
"assign_student_loan_plan_cohorts",
"assign_uniform_draw",
"aggregate_person_to_benunit",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5759,6 +5759,34 @@
}
}
},
{
"type": "object",
"additionalProperties": false,
"required": [
"kind",
"output",
"contract_key",
"period_source",
"dtype"
],
"properties": {
"kind": {
"const": "assign_period_constant"
},
"output": {
"type": "string"
},
"contract_key": {
"type": "string"
},
"period_source": {
"const": "build_year"
},
"dtype": {
"const": "float64"
}
}
},
{
"type": "object",
"additionalProperties": false,
Expand Down
Loading
Loading