perf(csr): sparse shift/roll/diff and merge(dim=...); densify notice on by default in sparse models

[FabianHofmann]

Closes #1011. Part of #969 and #972 (phase 2, PR 8). Stacked on #1014: until it merges, this PR also shows its commits; only the last commit belongs here.

Note

The following content was generated by AI.

Changes proposed in this Pull Request

• Sparse shift, roll and diff. The shift runs on the grid's row numbers with the dense (xarray) semantics, and the rows are gathered from the CSR backing. Shifted-in cells are absent, as on the dense path. roll wraps around, roll_coords included; the constant and auxiliary coordinates follow. diff is self - self.shift(...) and so stays sparse too. Positional arguments work as on Dataset.shift/Dataset.roll.
• Sparse linopy.merge(..., dim=<grid dim>). Rows are stacked in input order. The result coordinates are what the dense path's xr.concat yields on coordinate-only grids: overlapping and duplicate labels along dim, the join of the other dims (cells the join creates are absent), the v1 label check of the auto-detected join and auxiliary coordinates. Dense operands and operands in a different dim order are converted onto the template order first. The operands stay sparse.
• One row-gather primitive. CSRLinearExpression.taken now also serves the permutation fast path of reindexed and the dim reordering of concatenated. concatenated raises if the stacked row count does not match the target grid.
• Densify notice on by default in sparse models. options["warn_on_densify"] defaults to None: implicit fallbacks warn in Model(sparse=True), nothing warns in a dense model. Explicit conversions (CSRConstraint.mutable()/to_dense(), add_constraints(freeze=False)) warn only when the option is True. Internal conversions such as Constraints.set_blocks count as implicit.
• LinearExpression.sel/isel/[]/shift/roll share one small _gathered helper.
• The suite-wide ignore of the notice moved from pyproject.toml to a module-level pytestmark in test/test_csr.py; no other test file emits it.

Behaviour changes

• A sparse model now emits a PerformanceWarning ("Sparse (CSR) backing densified: ...") on every implicit densification. Suites that run with -W error will fail on these until they set linopy.options["warn_on_densify"] = False or remove the fallback.
• options["warn_on_densify"] now defaults to None (was False). True also covers dense models and explicit conversions; False silences it.
• shift, roll, diff and merge(dim=<grid dim>) on a CSR-backed expression now return a CSR-backed expression (is_sparse is True).

Thanks to draft PR #1005 by @MaykThewessen for the approach of letting xr.concat on coordinate-only grids define the result coordinates, and for test scenarios.

Operations that still densify (with a notice)

• shift with a fill_value.
• merge along a new dimension, with extra xr.concat arguments, or over non-unique labels that need aligning.
• merge of operands over different dimensions, i.e. broadcasting such as bal + s with s over a subset of the dims of bal.
• sel/isel/where that introduce dimensions or touch MultiIndex labels; where with a callable condition or a non-scalar other.
• Elementwise operations with a constant over new dimensions; the @ edge cases (zero-dim, non-unique or MultiIndex labels, zero-size grid).
• .data reads, e.g. by any operation without a sparse path, and groupby(...).sum(use_fallback=True).

Timings (2000 x 720 grid, groupby with uneven group sizes, 800 terms wide; tracemalloc on)

operation	before	after
shift(t=1)	1069 ms, 4.6 GB	12 ms, 56 MB
diff("t")	2626 ms, 9.2 GB	89 ms, 174 MB
merge([...], dim="t")	436 ms, 2.8 GB	19 ms, 61 MB

"After" re-measured on the final commit. The taken refactor moves merge(dim) from 58 to 61 MB peak; times are unchanged within noise.

Checklist

• AI-generated content is marked (see AGENTS.md).
• Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
• Unit tests for new features were added (if applicable).
• A note for the release notes doc/release_notes.rst of the upcoming release is included.
• I consent to the release of this PR's code under the MIT license.

[github-actions]

Build cost — v1 vs legacy

v1 build peak & time relative to legacy, on this commit — not a comparison against master (that is CodSpeed).

peak — v1 / legacy	time — v1 / legacy

Full table (time + peak, mean)

benchmarks/drivers/test_build.py::test_build[basic-n=10]
                 time (s)         peak (KiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.0899 (1.05)   │   15.03 (1.00) 
 (v1)        0.0856 (1.0)   │    15.00 (1.0) 

benchmarks/drivers/test_build.py::test_build[basic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.09973 (1.09)   │   12.04 (1.00) 
 (v1)        0.09111 (1.0)   │    12.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=0]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03919 (1.07)   │   15.20 (1.0) 
 (v1)        0.03676 (1.0)   │   15.20 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=100]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.05843 (1.13)   │   44.93 (1.0) 
 (v1)        0.05159 (1.0)   │   44.93 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04333 (1.07)   │   11.51 (1.0) 
 (v1)         0.0404 (1.0)   │   11.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=10]
                 time (s)         peak (KiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1031 (1.05)   │   24.34 (1.06) 
 (v1)       0.09781 (1.0)   │    23.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=250]
                 time (s)         peak (MiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1131 (1.04)   │   16.12 (1.00) 
 (v1)         0.109 (1.0)   │    16.12 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=10000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.02459 (1.04)   │   752.18 (1.10) 
 (v1)        0.02355 (1.0)   │    685.15 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=100]
                 time (s)        peak (KiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.0242 (1.04)   │   3.12 (1.33) 
 (v1)       0.02327 (1.0)   │    2.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=0]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.06588 (1.19)   │   126.16 (1.44) 
 (v1)        0.05514 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=100]
                 time (s)          peak (MiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.0652 (1.21)   │   126.16 (1.44) 
 (v1)       0.05411 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=50]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.06631 (1.21)   │   126.16 (1.44) 
 (v1)        0.05474 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[masked-n=100]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)    0.05647 (1.0)   │    715.12 (1.0) 
 (v1)       0.05727 (1.01)   │   787.73 (1.10) 

benchmarks/drivers/test_build.py::test_build[masked-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.05466 (1.06)   │   4.54 (1.27) 
 (v1)        0.05147 (1.0)   │    3.57 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=0]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.3829 (1.02)   │   704.12 (1.09) 
 (v1)        0.3769 (1.0)   │    643.85 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=100]
                time (s)        peak (MiB) 
 name               mean   │          mean 
───────────────────────────────────────────
 (legacy)   0.404 (1.04)   │   18.34 (1.0) 
 (v1)        0.388 (1.0)   │   18.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=50]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.4059 (1.03)   │   9.54 (1.0) 
 (v1)        0.3944 (1.0)   │   9.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.07729 (1.06)   │   3.77 (1.12) 
 (v1)        0.07266 (1.0)   │    3.37 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=50]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.07667 (1.05)   │   216.59 (1.10) 
 (v1)        0.07281 (1.0)   │    196.23 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=0]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.03956 (1.07)   │   938.49 (1.0) 
 (v1)          0.037 (1.0)   │   938.49 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.04066 (1.08)   │   9.66 (1.0) 
 (v1)        0.03753 (1.0)   │   9.66 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.04042 (1.08)   │   5.32 (1.0) 
 (v1)        0.03742 (1.0)   │   5.32 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=0]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.0202 (1.02)   │   1.28 (1.0) 
 (v1)        0.0199 (1.0)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.02026 (1.01)   │   1.28 (1.0) 
 (v1)        0.02004 (1.0)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)    0.01987 (1.0)   │   1.28 (1.0) 
 (v1)       0.02003 (1.01)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=1000]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.1941 (1.04)   │   946.85 (1.06) 
 (v1)        0.1867 (1.0)   │    891.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=10]
                 time (s)         peak (KiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1909 (1.04)   │   12.01 (1.00) 
 (v1)        0.1839 (1.0)   │    11.99 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.05046 (1.08)   │   147.70 (1.06) 
 (v1)        0.04679 (1.0)   │    139.87 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04889 (1.06)   │   2.60 (1.09) 
 (v1)        0.04601 (1.0)   │    2.38 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=0]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04079 (1.07)   │   696.75 (1.03) 
 (v1)         0.0383 (1.0)   │    673.70 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=100]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)    0.08642 (1.0)   │   137.97 (1.0) 
 (v1)       0.08677 (1.00)   │   137.97 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.06313 (1.05)   │   69.22 (1.0) 
 (v1)        0.06004 (1.0)   │   69.22 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04886 (1.14)   │   402.33 (1.00) 
 (v1)        0.04268 (1.0)   │    402.30 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04659 (1.10)   │   3.19 (1.19) 
 (v1)        0.04252 (1.0)   │    2.69 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.05079 (1.05)   │   29.00 (1.54) 
 (v1)        0.04831 (1.0)   │    18.84 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.06166 (1.05)   │   37.95 (1.43) 
 (v1)        0.05885 (1.0)   │    26.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[storage-n=10]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.09727 (1.0)   │    410.93 (1.0) 
 (v1)       0.1007 (1.03)   │   427.84 (1.04) 

benchmarks/drivers/test_build.py::test_build[storage-n=250]
                 time (s)         peak (MiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)    0.1049 (1.0)   │     9.94 (1.0) 
 (v1)       0.1079 (1.03)   │   10.22 (1.03) 

📊 Interactive plots + CSV: download the semantics-report-v1-vs-legacy artifact from this run.

_{Report-only · not a gate · refreshed on every push · obsolete once legacy is dropped.}

[codspeed]

Merging this PR will not alter performance

✅ 181 untouched benchmarks
⏩ 181 skipped benchmarks¹

---

_{Comparing perf/densify-gaps-1011 (bf79da4) with master (69dd5c9)²}

Footnotes

1. 181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. No successful run was found on master (0611af7) during the generation of this report, so 69dd5c9 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩