You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[BUG] Fitting a Triangle with a virtual column gives NaN for that column, with no warning #1607
When a Triangle has a virtual column, fitting the whole Triangle gives NaN for that column with no warning, while fitting the column on its own works. On raa:
1981-01-01 1982-01-01 1983-01-01 ... 1988-01-01 1989-01-01 1990-01-01
2261 NaN NaN NaN ... NaN NaN NaN
[1 rows x 10 columns]
106561.11413060508
__setitem__ stores an all-NaN placeholder slice for a callable (slice.py L894–L895), and tri["rest"] evaluates the function. A fit of the whole Triangle uses the placeholder: Development().fit(tri).ldf_["rest"] and tri.link_ratio["rest"] are all NaN, and MackChainladder gives the same NaN ultimate. ultimate_ and ldf_ list no virtual columns (methods/base.py L43 clears them on the ultimate), so only X_.virtual_columns shows that rest was virtual. I checked the default numpy backend only. On the sparse backend, __setitem__ creates no placeholder.
Expected: either the column is evaluated before fitting,
or fit raises or warns that virtual columns aren't supported. The user guide says virtual columns defer the calculation "to when needed", so I'd lean toward evaluating them in fit. I can open a PR with whichever you prefer.
(This issue was written with Árbol, an AI assistant, under my direction.)
Agreed. tri.age_to_age["rest"] is the same placeholder: on current main (46727b1) it is all NaN, while tri["rest"].age_to_age gives the correct ratios. It also means a fix has to evaluate the column on the cumulative triangle before link ratios are taken, since x["values"] - x["half"] on the ratio triangle gives 0 (both columns have the same ratios).
Given the size, would a small first PR help, where fit raises (or warns) when the triangle has virtual columns, with evaluation before link_ratio and fit in a later PR? The sparse backend creates no placeholder, so it needs its own check either way. I'm happy to take either part.
(This comment was written with Árbol, an AI assistant, under my direction.)
Agreed on a small first PR. values is a plain attribute that the library itself reads and assigns in many places (common.py, pandas.py, slice.py, triangle.py), so a warning there would also fire on internal reads. I'd warn where a calculation starts instead: in link_ratio (which age_to_age aliases) and in estimator fit, whenever the triangle carries virtual columns. I'll add the disclaimer next to the "should only reference other columns" note in the Virtual Columns section of docs/user_guide/triangle.ipynb.
With the disclaimer in place, do you still want a later PR that evaluates virtual columns before fitting, or should they stay for display and export only?
(This comment was written with Árbol, an AI assistant, under my direction.)
ah, right, we never shimmed values. in theory, if we refactor values into a shim, with an eval on all virtual columns, does that get us most of the way there?
we can start with a docs-only PR that warns virtual columns only have limited support.
Mostly, I think: reads would see evaluated numbers, so age_to_age, link_ratio and fit should work on virtual columns. A shim has to handle three things:
In-place writes. Some code writes through .values (munich.py, incremental.py, methods/base.py, clark.py, triangle_weight.py, and values.shape = ... in slice.py and triangle.py). If the getter returns a freshly evaluated array, those writes are lost.
Recursion. A virtual column's lambda reads other columns through __getitem__, which reads values, so the getter needs a re-entrancy guard.
Cost. The library reads .values constantly, so running every lambda on every read needs a cache that is cleared when a column changes.
to_frame already evaluates virtual columns its own way (row by row, after fillna(0)), so the shim should match it or replace that path.
I'll start with the docs-only PR. I can prototype the shim afterwards and run the full test suite before proposing it.
(This comment was written with Árbol, an AI assistant, under my direction.)
some of the prior conversations are coming back to me now. at one point we were discussing a migration to polars. but that haven't been a high priority for a while. so there's no rush to make the package fully virtual column compatible.
Activity
henrydingliu commentedon Nov 15, 2026
this feels pretty big. tri.age_to_age["rest"] looks funny too
scks1 commentedon Nov 15, 2026
Agreed.
tri.age_to_age["rest"]is the same placeholder: on current main (46727b1) it is all NaN, whiletri["rest"].age_to_agegives the correct ratios. It also means a fix has to evaluate the column on the cumulative triangle before link ratios are taken, sincex["values"] - x["half"]on the ratio triangle gives 0 (both columns have the same ratios).Given the size, would a small first PR help, where
fitraises (or warns) when the triangle has virtual columns, with evaluation beforelink_ratioandfitin a later PR? The sparse backend creates no placeholder, so it needs its own check either way. I'm happy to take either part.(This comment was written with Árbol, an AI assistant, under my direction.)
henrydingliu commentedon Nov 15, 2026
good with a small first PR.
fitdoesn't feel like sufficiently upstream ifage_to_ageis failing. isvaluesa better warning spot?we should also add a disclaimer into the docs that virtual columns can't be used further an any property or estimation.
scks1 commentedon Nov 16, 2026
Agreed on a small first PR.
valuesis a plain attribute that the library itself reads and assigns in many places (common.py,pandas.py,slice.py,triangle.py), so a warning there would also fire on internal reads. I'd warn where a calculation starts instead: inlink_ratio(whichage_to_agealiases) and in estimatorfit, whenever the triangle carries virtual columns. I'll add the disclaimer next to the "should only reference other columns" note in the Virtual Columns section ofdocs/user_guide/triangle.ipynb.With the disclaimer in place, do you still want a later PR that evaluates virtual columns before fitting, or should they stay for display and export only?
(This comment was written with Árbol, an AI assistant, under my direction.)
henrydingliu commentedon Nov 16, 2026
ah, right, we never shimmed
values. in theory, if we refactor values into a shim, with an eval on all virtual columns, does that get us most of the way there?we can start with a docs-only PR that warns virtual columns only have limited support.
scks1 commentedon Nov 16, 2026
Mostly, I think: reads would see evaluated numbers, so
age_to_age,link_ratioandfitshould work on virtual columns. A shim has to handle three things:.values(munich.py,incremental.py,methods/base.py,clark.py,triangle_weight.py, andvalues.shape = ...inslice.pyandtriangle.py). If the getter returns a freshly evaluated array, those writes are lost.__getitem__, which readsvalues, so the getter needs a re-entrancy guard..valuesconstantly, so running every lambda on every read needs a cache that is cleared when a column changes.to_framealready evaluates virtual columns its own way (row by row, afterfillna(0)), so the shim should match it or replace that path.I'll start with the docs-only PR. I can prototype the shim afterwards and run the full test suite before proposing it.
(This comment was written with Árbol, an AI assistant, under my direction.)
henrydingliu commentedon Nov 16, 2026
some of the prior conversations are coming back to me now. at one point we were discussing a migration to polars. but that haven't been a high priority for a while. so there's no rush to make the package fully virtual column compatible.
[DOCS] Keep the virtual-column warning to its first two sentences
scks1 commentedon Nov 16, 2026
Makes sense. I'll hold off on the
valuesshim prototype and keep #1632 to the docs warning.I can leave #1607 open to track full virtual column support, or close it once #1632 merges, whichever you prefer.
(This comment was written with Árbol, an AI assistant, under my direction.)
henrydingliu commentedon Nov 17, 2026
let's keep this issue around. i expect to get more inquiries regarding virtual columns as pd.col gets more traction.
[DOCS] Warn that virtual columns have limited support
[DOCS] Keep the virtual-column warning to its first two sentences