Repository navigation
Description
TweedieGLM and BarnettZehnwirth accept a design matrix with fewer independent columns than columns and fit it without a warning. The common way to get there is to add a calendar term to an origin and development design. In the frame these estimators build, valuation is a linear function of origin and development in every cell, so the three columns are linearly dependent. Kuang, Nielsen and Nielsen describe this identification problem for the extended chain-ladder model in "Identification of the age-period-cohort model and the extended chain-ladder model", Biometrika 95(4), 2008, pages 979 to 986, https://doi.org/10.1093/biomet/asn026.
On main at cc3335c, with alpha=0 now that #1663 passes it through:
import numpy as np
import chainladder as cl
gen = cl.load_sample("genins")
for f in ["C(development) + C(origin)",
"C(development) + C(origin) + valuation",
"development + origin + valuation"]:
m = cl.TweedieGLM(design_matrix=f, alpha=0, max_iter=5000, tol=1e-10).fit(gen)
X = np.asarray(m.model_.estimator_ml.named_steps["design_matrix"].transform(m.model_.df_))
ult = cl.Chainladder().fit(m.transform(gen)).ultimate_.sum()
print(f, X.shape[1], np.linalg.matrix_rank(X), round(ult, 2))The loop prints these values.
design columns rank total ultimate
C(development) + C(origin) 19 19 53,038,954.62
C(development) + C(origin) + valuation 20 19 53,038,910.06
development + origin + valuation 4 3 55,442,708.27
Volume-weighted chain ladder gives 53,038,945.61. No warning appears in any of the three fits.
The forecasts hold steady because the same linear relation among origin, development and valuation also holds in the future cells. The companion paper by the same authors, "Forecasting with the age-period-cohort model and the extended chain-ladder model", Biometrika 95(4), 2008, pages 987 to 991, sets out when forecasts stay invariant. The coefficients carry no meaning on their own. In the linear design the fit returns 4.51786 for origin and -4.48402 for valuation, and the two nearly cancel.
BarnettZehnwirth shows the same thing on abc, and here one reported statistic changes.
formula columns rank mse_resid_ total ultimate
C(origin)+C(development) 21 21 0.006842 15,447,587.43
C(origin)+C(development)+valuation 22 21 0.006997 15,447,587.43
The ultimate does not move, but mse_resid_ divides by 66 cells minus 22 coefficients where the design has only 21 independent columns, so mse_resid_ rises by a factor of 45/44 and std_residuals_ shrink by its square root.
I suggest a check at fit time in both estimators that compares np.linalg.matrix_rank of the design with its column count and raises a warning naming the formula when they differ. BarnettZehnwirth could also use the rank for the residual degrees of freedom. The change is small and self-contained, so it would suit a new contributor.
(This issue was written with Árbol, an AI assistant, under my direction.)
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Effort
Scope
Projects
- StatusShow more project fieldsReady
Activity
henrydingliu commentedon Nov 19, 2026
is this along the same lines as the warning added in commit 9fe6fec? i don't recall why i removed it.
scks1 commentedon Nov 19, 2026
Yes, it aims at the same problem. #642 gives the reason the warning went away. @danielfong-act wrote that "any formula constructed by PTF_formula will trigger it." The 9fe6fec check counted formula terms without
C(, and the trend segments thatPTF_formulabuilds are all such terms.Counting terms misses in both directions. On
abcat cc3335c, the categorical design with a calendar term slips past the count, and a full-rank PTF design trips it. A rank check gets both right.Snippet for the table
old_9fe6fec_countcopies the loop that 9fe6fec added toBarnettZehnwirth.fit.A warning that compares
np.linalg.matrix_rankof the design with its column count stays silent on PTF formulas, so it would not run into the problem that removed the old one. I'll leave the change for a new contributor.(This comment was written with Árbol, an AI assistant, under my direction.)