Add space filling curve (SFC) and hybrid SFC/graph partitioning algorithms for building parallel meshes

[garth-wells]

Summary

Adds space-filling curve (SFC) and hybrid SFC/graph partitioning, so that mesh construction cost can stay roughly flat as the rank count grows, instead of scaling with the (increasingly expensive) cost of graph partitioning. Also includes a general mesh-construction performance pass, a ParMETIS re-partitioner for rebalancing, and demo_partition.py, a new demo comparing all available partitioners on the same input under varying degrees of input-distribution randomness.

What's new

Geometric and hybrid partitioning types (graph/partition.h)

• graph::partition_fn — graph-only partitioners (required dual graph, no coordinates).
• graph::geom_partition_fn — partitioners that use node positions and optionally the graph (e.g. for ghosting only); the graph argument is std::optional.
• graph::hybrid_partition_fn — partitioners that need both the graph and node positions unconditionally, because the graph edges are part of the partitioning decision itself, not only used for ghosting.

Space-filling curve partitioning (graph/sfc.h{cpp})

• graph::partition_sfc_morton / partition_sfc_hilbert: partition points along a Morton or Hilbert curve. No external library required. Cost is nearly independent of rank count, at the price of a larger edge cut than graph partitioning.
• graph::sfc::partitioner: wraps either curve as a geom_partition_fn.

ParMETIS additions (graph/partitioners.h{cpp})

• graph::parmetis::geom_partitioner (ParMETIS_V3_PartGeom) and geom_partitioner_kway (ParMETIS_V3_PartGeomKway, a hybrid_partition_fn since it uses the graph edges as well as the coordinates).
• graph::parmetis::repartitioner (ParMETIS_V3_AdaptiveRepart): treats the current data distribution as the current partition and computes a new one that balances load while limiting how many nodes move — suited to rebalancing after non-uniform refinement, as opposed to partitioning from scratch.
• Default PT-SCOTCH strategy changed to strategy::speed (see performance data below).

Mesh-level wrappers (mesh/partition.h{cpp})

• mesh::create_geometric_cell_partitioner: computes cell centroids and applies a geom_partition_fn; builds the mesh dual graph only when ghosting is requested (skipped otherwise, since a purely geometric partitioner does not need it for the partitioning decision itself).
• mesh::create_hybrid_cell_partitioner: as above but for a hybrid_partition_fn — always builds and supplies the dual graph, since these partitioners need it unconditionally.
• Split out of mesh/utils.h{cpp} into their own files.

create_mesh performance and structure

• Removed redundant copies/sorts in the serial path (measured 4.29 s → 3.48 s for 6M tetrahedra, Release build, single rank): avoided a second copy of the concatenated cell array, reused the sorted unique vertex indices that topology construction already computes instead of re-deriving them, and skipped the (identity) topology-extraction step for 'P1 geometry'.
• Extracted the cell-partitioning/redistribution logic into mesh::impl::partition_cells, shrinking create_mesh to a clearer sequence of named steps.

demo_partition.py

• Compares every available partitioner (ParMETIS Kway/GeomKway/Geom, PT-SCOTCH, SFC Morton/Hilbert) on the same distributed input, reporting cell imbalance and edge cut, across input distributions ranging from fully locality-preserving to fully randomised.
• A "two-stage partitioning" comparison: partition with the space-filling curve first (cheap, insensitive to input distribution), then re-partition the result with a graph partitioner (now running on a well-distributed input) — the general idea ParMETIS GeomKway embodies, but applicable around any graph partitioner.

Performance data

• create_mesh copy/sort removal: 4.29 s → 3.48 s, 6M tetrahedra, serial, Release build.
• SFC vs. graph partitioning for mesh construction: 1.48x faster at 40 ranks for 15.4M tetrahedra, at the cost of 24% more ghost cells.
• PT-SCOTCH strategy::speed vs. default: ~20% faster with no measurable change in cut edges (20 ranks/3.1M cells: 0.88–0.99 s → 0.69–0.77 s, 148.8k → 149.5k cut edges; 40 ranks/6M cells: 2.36 s → 2.19 s, 329.6k → 328.3k cut edges). strategy::quality was measured slower and worse (more cut edges) than the default.
• ParMETIS Kway vs. GeomKway crossover (n=128, 12.6M tetrahedra, 20 ranks): with locality-preserving input, Kway is faster (2.35 s vs. 2.58 s); with fully randomised input, GeomKway wins (4.78 s vs. 5.99 s) — Kway degrades 2.55x under randomisation against GeomKway's 1.85x.
• ParMETIS re-partitioner vs. partitioning from scratch (20 ranks, 3.1M cells, deliberately skewed starting distribution, imbalance 3.5): re-partitioning rebalances to 1.014 while moving 74.5% of cells, against 81.8% moved for partitioning from scratch, at the cost of a 7% larger edge cut.
• Two-stage SFC-then-PT-SCOTCH vs. PT-SCOTCH alone, fully randomised input:
  • 12.6M tetrahedra, 20 ranks: time inside SCOTCH falls from 9.52 s to 2.25 s (4.2x), edge cut slightly smaller (385506 → 375596); end-to-end mesh creation 11.7 s → 7.1 s.
  • 42.5M tetrahedra, 20 ranks (measured separately): SCOTCH alone 46.6 s total (38.1 s inside SCOTCH) vs. two-stage 24.2 s total (14.1 s inside SCOTCH) — SCOTCH itself is ~2.7x faster once the input has been pre-localised, edge cut essentially a wash (859442 vs. 853204, within PT-SCOTCH's own run-to-run variability).
  • At 50% (rather than fully) randomised input, 42.5M tetrahedra: SCOTCH alone 36.4 s vs. two-stage 24.2 s; the time advantage shrinks with less randomness, as expected, since there is less locality to lose in the first place.

Scaling data (demo_partition.py)

Rank-count scaling, fixed mesh, fully randomised input

82,944 tetrahedra, fraction=1.0 (fully randomised). Time in seconds; edge cut in parentheses:

Ranks	ParMETIS Kway	ParMETIS GeomKway	ParMETIS Geom	PT-SCOTCH	SFC Morton	SFC Hilbert
1	0.085 (0)	0.098 (0)	0.100 (0)	0.086 (0)	0.111 (0)	0.110 (0)
2	0.094 (2760)	0.096 (2556)	0.077 (2390)	0.102 (2344)	0.074 (2364)	0.080 (2464)
3	0.080 (4466)	0.074 (4186)	0.046 (5808)	0.126 (4052)	0.038 (5828)	0.043 (5570)
5	0.060 (6756)	0.055 (6470)	0.044 (8684)	0.111 (5688)	0.037 (8660)	0.040 (7188)

(At 1 rank there is nothing to cut, hence 0.) At this mesh size, absolute times are dominated by fixed per-call overhead rather than by the cost of partitioning itself, so this mainly demonstrates correct behaviour at low rank counts. It is not a demonstration of asymptotic scaling — see below and the two-stage numbers in Performance data above for where the scaling advantage of the SFC/hybrid approach actually shows up.

Problem-size scaling at fixed rank count (20 ranks, fully randomised input)

Tetrahedra	ParMETIS Kway	ParMETIS GeomKway	ParMETIS Geom	PT-SCOTCH	SFC→PT-SCOTCH (two-stage)	SFC Morton	SFC Hilbert
12.6M	5.82 s (436382)	4.61 s (434496)	3.41 s (509306)	11.82 s (385506)	7.96 s (382700)	3.24 s (510414)	3.30 s (458188)
42.5M	21.04 s (985510)	16.43 s (974388)	11.94 s (1144526)	41.02 s (853204)	26.19 s (859442)	11.48 s (1148982)	12.04 s (1028312)
time ratio for 3.4x more cells	3.62x	3.57x	3.50x	3.47x	3.29x	3.54x	3.65x

At fixed rank count, all partitioners scale roughly linearly with cell count over this range (3.3–3.65x time for a 3.4x increase in cells) — none is dramatically worse than linear here. PT-SCOTCH remains the most expensive in absolute terms at both sizes, by a wide margin; the two-stage route is consistently faster than PT-SCOTCH alone at both sizes (7.96 s vs. 11.82 s, and 26.19 s vs. 41.02 s) and also has the lowest time ratio of the seven columns (3.29x vs. PT-SCOTCH alone's 3.47x), i.e. it scales somewhat more mildly with problem size here, not just faster in absolute terms — consistent with the SFC stage taming PT-SCOTCH's own scale-sensitive coarsening cost rather than only offering a one-off saving. The rank-count-independent cost advantage that motivates the SFC/hybrid approach more broadly is not fully visible in a same-rank-count comparison like this one, and would be expected to grow more pronounced at higher rank counts than tested here.

Known follow-up

Redistributing cells for the two-stage comparison currently goes through building a full intermediate mesh, most of whose cost (topology, geometry, ghost cells) is wasted since only the cell destinations are needed before the second stage. A redistribute_by_partitioner helper that calls the partitioner directly and exchanges only the cell rows was prototyped in the demo and measured at roughly half the cost of the full-mesh route for this step, but is not yet part of the public API.

Filed #4402: for mixed-topology meshes, cell redistribution currently pays for one NBX neighbourhood-discovery round per cell type; the destination rank sets for different cell types typically overlap substantially, so this is a candidate for future optimisation. Does not affect single-cell-type meshes, which are unaffected (one cell type, one round).

Testing

C++ unit tests pass on 1, 2 and 3 ranks; Python test suite passes; demo_partition.py runs via the demo test harness across multiple rank counts and mesh sizes, including a 42.5M-tetrahedron mesh on 20 ranks.