perf(csr): one-pass n-ary merge, row gather for permuting reindex by FabianHofmann · Pull Request #1014 · PyPSA/linopy

FabianHofmann · 2026-10-06T14:00:43Z

Closes #1010. Part of #972 (phase 2, PR 7).

Note

The following content was generated by AI.

Changes proposed in this Pull Request

One-pass n-ary merge. CSRLinearExpression.added(self, *others) now takes any number of operands on the same grid and builds the sum in a single COO pass. _try_csr_merge calls it once instead of folding pairwise. Explicit zero coefficients and absent cells keep their current behaviour. Auxiliary coordinates are united, earlier operands take precedence, and conflicting ones still raise.
Row gather for permuting reindex. When reindexed only permutes the existing cells (no dropped and no new labels), it gathers the CSR rows and constants directly instead of going through COO.
Index dtype in aggregated. The aggregated row indices are cast to the store's index dtype before the COO to CSR conversion.

The added(self, *others) change is byte-identical to the one in open PR #992, so the two PRs merge without conflict.

Chained + is unchanged, because + is binary.

Benchmark (scratch script dev-scripts/bench_1010.py, before → after)

case	before	after
merge of 3 sparse operands	118 ms	84 ms
transposed merge	58 ms	45 ms
permuting reindex	16 ms	4 ms
chained `+`	unchanged	unchanged

Checklist

AI-generated content is marked (see AGENTS.md).
Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
Unit tests for new features were added (if applicable).
A note for the release notes doc/release_notes.rst of the upcoming release is included.
I consent to the release of this PR's code under the MIT license.

) Make CSRLinearExpression.added n-ary in one COO pass (as in #992), gather rows directly when reindexed only permutes cells, and index aggregated rows in the store's index dtype.

github-actions · 2026-10-06T14:04:52Z

Build cost — v1 vs legacy

v1 build peak & time relative to legacy, on this commit — not a comparison against master (that is CodSpeed).

peak — v1 / legacy	time — v1 / legacy

Full table (time + peak, mean)

benchmarks/drivers/test_build.py::test_build[basic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.08017 (1.12)   │   15.03 (1.00) 
 (v1)        0.07137 (1.0)   │    15.00 (1.0) 

benchmarks/drivers/test_build.py::test_build[basic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.08843 (1.11)   │   12.04 (1.00) 
 (v1)        0.07999 (1.0)   │    12.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=0]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03379 (1.08)   │   15.20 (1.0) 
 (v1)        0.03132 (1.0)   │   15.20 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=100]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.05502 (1.07)   │   44.93 (1.0) 
 (v1)        0.05143 (1.0)   │   44.93 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03816 (1.07)   │   11.51 (1.0) 
 (v1)        0.03568 (1.0)   │   11.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=10]
                 time (s)         peak (KiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.0862 (1.05)   │   24.34 (1.06) 
 (v1)       0.08205 (1.0)   │    23.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.09951 (1.06)   │   16.12 (1.00) 
 (v1)        0.09384 (1.0)   │    16.12 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=10000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.02089 (1.06)   │   752.18 (1.10) 
 (v1)        0.01962 (1.0)   │    685.15 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=100]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.02094 (1.10)   │   3.12 (1.33) 
 (v1)        0.01909 (1.0)   │    2.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=0]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.07156 (1.26)   │   126.16 (1.44) 
 (v1)        0.05701 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=100]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.06948 (1.25)   │   126.16 (1.44) 
 (v1)        0.05541 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=50]
                 time (s)          peak (MiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.0707 (1.26)   │   126.16 (1.44) 
 (v1)       0.05619 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[masked-n=100]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04815 (1.03)   │    715.12 (1.0) 
 (v1)         0.0466 (1.0)   │   787.73 (1.10) 

benchmarks/drivers/test_build.py::test_build[masked-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04663 (1.11)   │   4.54 (1.27) 
 (v1)        0.04213 (1.0)   │    3.57 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=0]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.3278 (1.04)   │   704.12 (1.09) 
 (v1)        0.3146 (1.0)   │    643.85 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=100]
                 time (s)        peak (MiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.3444 (1.04)   │   18.34 (1.0) 
 (v1)        0.3318 (1.0)   │   18.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=50]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.3412 (1.04)   │   9.54 (1.0) 
 (v1)        0.3277 (1.0)   │   9.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.06565 (1.09)   │   3.77 (1.12) 
 (v1)         0.0603 (1.0)   │    3.37 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=50]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.06589 (1.08)   │   216.59 (1.10) 
 (v1)        0.06073 (1.0)   │    196.23 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=0]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.03428 (1.10)   │   938.49 (1.0) 
 (v1)        0.03107 (1.0)   │   938.49 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=100]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.0354 (1.08)   │   9.66 (1.0) 
 (v1)       0.03291 (1.0)   │   9.66 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.03476 (1.08)   │   5.32 (1.0) 
 (v1)        0.03233 (1.0)   │   5.32 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=0]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.01804 (1.05)   │   1.28 (1.0) 
 (v1)        0.01722 (1.0)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)    0.01716 (1.0)   │   1.28 (1.0) 
 (v1)       0.01728 (1.01)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)    0.01739 (1.0)   │   1.28 (1.0) 
 (v1)       0.01741 (1.00)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=1000]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.1638 (1.05)   │   946.85 (1.06) 
 (v1)        0.1562 (1.0)   │    891.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=10]
                 time (s)         peak (KiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1617 (1.06)   │   12.01 (1.00) 
 (v1)        0.1532 (1.0)   │    11.99 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04186 (1.07)   │   147.70 (1.06) 
 (v1)        0.03922 (1.0)   │    139.87 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04152 (1.06)   │   2.60 (1.09) 
 (v1)        0.03917 (1.0)   │    2.38 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=0]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03499 (1.07)   │   696.75 (1.03) 
 (v1)        0.03273 (1.0)   │    673.70 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=100]
                 time (s)         peak (MiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1009 (1.04)   │   137.97 (1.0) 
 (v1)       0.09727 (1.0)   │   137.97 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)    0.06284 (1.0)   │   69.22 (1.0) 
 (v1)       0.06358 (1.01)   │   69.22 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=1000]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.0399 (1.10)   │   402.33 (1.00) 
 (v1)       0.03613 (1.0)   │    402.30 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03908 (1.10)   │   3.19 (1.19) 
 (v1)         0.0355 (1.0)   │    2.69 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.04352 (1.07)   │   29.00 (1.54) 
 (v1)        0.04049 (1.0)   │    18.84 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=250]
                 time (s)         peak (MiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.0571 (1.13)   │   37.95 (1.43) 
 (v1)       0.05061 (1.0)   │    26.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[storage-n=10]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)    0.08273 (1.0)   │    410.93 (1.0) 
 (v1)       0.08451 (1.02)   │   427.84 (1.04) 

benchmarks/drivers/test_build.py::test_build[storage-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)    0.09147 (1.0)   │     9.94 (1.0) 
 (v1)       0.09332 (1.02)   │   10.22 (1.03)

📊 Interactive plots + CSV: download the semantics-report-v1-vs-legacy artifact from this run.

_{Report-only · not a gate · refreshed on every push · obsolete once legacy is dropped.}

codspeed · 2026-10-06T14:06:42Z

Merging this PR will improve performance by 44.36%

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 14 improved benchmarks
✅ 167 untouched benchmarks
⏩ 181 skipped benchmarks¹

Performance Changes

	Benchmark	`BASE`	`HEAD`	Efficiency
⚡	`test_to_lp[nodal_balance_sparse-severity=0]`	3.9 MB	2.5 MB	+54.8%
⚡	`test_to_lp[nodal_balance_sparse-severity=100]`	3.9 MB	2.5 MB	+54.71%
⚡	`test_to_lp[nodal_balance_sparse-severity=50]`	3.9 MB	2.5 MB	+54.71%
⚡	`test_to_lp[knapsack-n=10000]`	2.1 MB	1.4 MB	+52.91%
⚡	`test_to_lp[storage-n=250]`	38.5 MB	25.8 MB	+49.12%
⚡	`test_to_lp[merge_balance-severity=0]`	2.7 MB	1.8 MB	+48.43%
⚡	`test_to_lp[rolling-severity=0]`	2 MB	1.4 MB	+48.07%
⚡	`test_to_lp[nodal_balance-severity=0]`	3.8 MB	2.6 MB	+45.27%
⚡	`test_to_lp[nodal_balance-severity=50]`	3.8 MB	2.6 MB	+45.04%
⚡	`test_to_lp[storage-n=10]`	1.6 MB	1.1 MB	+44.89%
⚡	`test_to_lp[piecewise-n=1000]`	1.9 MB	1.4 MB	+37.3%
⚡	`test_to_lp[basic-n=250]`	29.9 MB	21.8 MB	+37.15%
⚡	`test_to_lp[expression_arithmetic-n=250]`	46.6 MB	35.3 MB	+32.01%
⚡	`test_to_lp[sos-n=1000]`	1.4 MB	1.1 MB	+21.19%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing perf/csr-kernel-1010 (d6da509) with master (7e007ff)²}

181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
No successful run was found on master (69dd5c9) during the generation of this report, so 7e007ff was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

FabianHofmann added 2 commits October 6, 2026 15:42

perf(csr): one-pass n-ary added, row gather for permuting reindex (#1010

9b43746

) Make CSRLinearExpression.added n-ary in one COO pass (as in #992), gather rows directly when reindexed only permutes cells, and index aggregated rows in the store's index dtype.

test: reuse densified helper for mypy; add release note (#1010)

a50c11b

This was referenced Oct 6, 2026

perf(csr): sparse shift/roll/diff and merge(dim=...); densify notice on by default in sparse models #1016

Merged

docs: document the sparse expression path #1017

Merged

FabianHofmann added 2 commits October 7, 2026 11:09

Merge branch 'master' into perf/csr-kernel-1010

7bab24f

test(csr): drop reindex test superseded by test_reindex_matches_dense

d6da509

FabianHofmann merged commit 0611af7 into master Oct 7, 2026
22 checks passed

FabianHofmann deleted the perf/csr-kernel-1010 branch October 7, 2026 09:40

Navigation Menu

Repository navigation

perf(csr): one-pass n-ary merge, row gather for permuting reindex - #1014

perf(csr): one-pass n-ary merge, row gather for permuting reindex#1014
FabianHofmann merged 4 commits into
masterfrom
perf/csr-kernel-1010

FabianHofmann commented Oct 6, 2026

Uh oh!

github-actions Bot commented Oct 6, 2026 •

edited

Loading

Uh oh!

codspeed Bot commented Oct 6, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

FabianHofmann commented Oct 6, 2026

Changes proposed in this Pull Request

Checklist

Uh oh!

github-actions Bot commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build cost — v1 vs legacy

Uh oh!

codspeed Bot commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will improve performance by 44.36%

Performance Changes

Footnotes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Oct 6, 2026 •

edited

Loading

codspeed Bot commented Oct 6, 2026 •

edited

Loading