Conversation
for more information, see https://pre-commit.ci
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1151 +/- ##
==========================================
+ Coverage 77.24% 77.42% +0.18%
==========================================
Files 63 64 +1
Lines 9378 9455 +77
Branches 1579 1585 +6
==========================================
+ Hits 7244 7321 +77
Misses 1532 1532
Partials 602 602
🚀 New features to boost your workflow:
|
| # Trusted but not installed | ||
| if canonical in TRUSTED_BACKENDS and get_backend(canonical) is None: | ||
| package = TRUSTED_BACKENDS[canonical]["package"] | ||
| raise ValueError( |
| # conformance test suite (squidpy.testing.backend_conformance). | ||
| TRUSTED_BACKENDS: dict[str, dict[str, Any]] = { | ||
| "rapids_singlecell": { | ||
| "aliases": ["rapids-singlecell", "rsc", "cuda", "gpu"], |
There was a problem hiding this comment.
why have multiple? maybe just have a error message helper that recognizes them and does “backend 'gpu' does not exist, did you mean 'rapids-singlecell'?”
There was a problem hiding this comment.
didn’t @selmanozleyen already build something like this? this looks like it’s similar code, so if the other version is merged, this should be unified with that.
There was a problem hiding this comment.
@selmanozleyen approach is similar but less extensible. It's still a draft PR like this one. This approach would immediately open the door for other backends. With 0 updates needed in squidpy.
| return wrapper | ||
|
|
||
|
|
||
| def _update_signatures() -> None: |
There was a problem hiding this comment.
This private function is not used in this file but it is exported. Therefore it shouldn't be private. The module _dispatch is already private so it is fine to remove the _ here
| def _get_param_sets( | ||
| func: Callable, | ||
| adapter_method: Callable, | ||
| func_name: str, |
There was a problem hiding this comment.
to be more clear func_name is not being used here
|
compared to #1093 I agree that renaming it to However, currently there is a blocker because the |
| Called once automatically after backend discovery so that ``help()`` / | ||
| IDE tooltips show the full parameter list (CPU + GPU + backend) with | ||
| documentation. |
There was a problem hiding this comment.
I am not sure about dynamically updating the signature and docstrings. I like the approach with just linking to the dispatched backend better in :#1093
@flying-sheep @ilan-gold wdyt? For example if I compiled the docs with gpu will it compile a mix of rsc and squidpy docs?
There was a problem hiding this comment.
I would ideally want to expose everything, even if dispatched to another function in the public function signature
| _update_signatures() | ||
|
|
||
|
|
||
| def _check_trusted(name: str) -> None: |
There was a problem hiding this comment.
again, should be check_trusted
I think But I think the right answer is to do what the rest of the pydata ecosystem does and just follow these patterns. I don't see any reason for us to stick out. Generally, I hope that a generalized version of this could be implemented into https://github.com/scverse/scverse-misc so that it would eventually be reused across all of our packages trivially. Edit: I just read on zulip:
Yeah SGTM but maybe |
But this way we can leave it to the user to either set the context with whatever backend of their choice. If we had jax backend for example we could do it either in cpu or gpu. |
| effective = local_backend or settings.backend | ||
|
|
||
| if effective == "cpu": | ||
| return func(*args, **kwargs) |
There was a problem hiding this comment.
This has a subtle bug if we want to generalize. We only bind *args by order and don't ever check if the order and the names match.
For example:
CPU: func(a, b)
GPU: func(b, a)
and both a and b are integers:
func(5, 10, backend="gpu")
becomes:
GPU sees b=5, a=10
If the backend computes something like a / b, you silently get 10 / 5 instead of 5 / 10. No exception, just incorrect output.
or imagine this
if you have
CPU: threshold, n_perms
GPU: n_perms, threshold
these won't error at the level they should. Maybe it will say n_perms should be an integer inside the function but it shouldn't have been dispatchable to begin with.
|
@Zethson We should also document the contract here. There are lots of assumptions hidden in this PR if we want to generalize. For example currently the dispatch is position based for args, see the review I did, if it's not addressed and generalized it will introduce subtle bugs. To expose these assumptions ideally we should have a For two functions
shared = [name for name in base_args if name in backend_arg_names]
backend_shared = [name for name in backend_args if name in base_arg_names]
# we want
shared == backend_sharedThese are the current terms for the contract. But instead of update-the-signature trick we can disallow |
Suggestion on top of #1284. Three bug fixes, one behaviour revert, and a restructuring that makes each niche flavor one call of a shared primitive. Bug fixes - The shell filter compared path *counts* against the visited set, so whether a pair was excluded depended on how many paths joined it -- grst's "not accounted for node degrees" in #1277. Worth noting the original CellCharter does not have this: its `_aggregate_neighbors` opens with `adj = adj.astype(bool)`, which makes the matmul a reachability test and `>` the logical and-not its comment claims. The port dropped that line. #1284's `frontier.data[:] = 1` fixes the unweighted case; on a weighted graph `1 > 0.5` still let a visited pair back in. - `n_hop_weights` shorter than `distance` logged an error and then raised `UnboundLocalError` on the unassigned `weights`. - The `calculate_niche` deprecation moved to `logg.warning`, which cannot be filtered, caught, or asserted on. Every other deprecation in squidpy uses `FutureWarning`, `_docs.py` documents that, and three test modules rely on it. Reverted - `_NhoodProfileEmbedder` was switched from matrix powers to disjoint shells while standardising on one hop helper. The two are not interchangeable: shells suit `concat`, where overlapping blocks would restate each other, and powers suit the weighted `sum`, where multiplicity *is* the distance weighting. `hop_mode` now carries both, defaulting per use, and `neighborhood` is bit-identical to main again at `distance > 1`. Restructuring - Hops now come from a numba breadth-first search. Each observation is reached once at its shortest distance, so the shells are disjoint by construction -- there is no visited set to subtract, and so no way for degree to leak in. 1.4-4.1x faster (1.06s -> 0.39s at 200k cells, 4 hops). - `nhood_aggregate` is the primitive all three flavors call, verified bit-identical to the embedders it replaces across 8 configurations. Unexposed: not in `gr/__init__.py` or `docs/api.md`. - Embedders are functions rather than classes; the `_NicheEmbedder` ABC is gone. A function is something backend dispatch can attach to, which a third-party subclass is not (#1151). - Sparse features stay sparse, as upstream CellCharter does, so the concatenated matrix -- `hops` times the width of the features -- is not densified before PCA. - `Array | CSBase` from fast_array_utils instead of `NDArrayA`, widening the embedding contract off numpy (#1245) while keeping the sparse half honest. Tests go 21 -> 46 in test_niche.py; every fix has one that fails without it.
Suggestion on top of #1284. Fixes - Shell exclusion compared path counts, not sets, so it depended on node degree (#1277). CellCharter does not have this: `_aggregate_neighbors` starts with `adj = adj.astype(bool)`, which makes `>` the logical and-not its comment claims, and the port dropped that line. `frontier.data[:] = 1` fixes only the unweighted case -- on a weighted graph `1 > 0.5` still readmits a visited pair. - Short `n_hop_weights` raised `UnboundLocalError` instead of failing cleanly. - The `calculate_niche` deprecation became `logg.warning`, which cannot be filtered, caught or asserted on; every other deprecation here uses `FutureWarning`. Revert - `neighborhood` was switched from matrix powers to disjoint shells. The two are not interchangeable: rings suit concat, where blocks must not restate each other, and powers suit the weighted sum, where multiplicity is the distance weighting. It is bit-identical to main again at distance > 1. Restructure - A numba breadth-first search reaches each cell once at its shortest distance, so the rings are disjoint by construction and there is no visited set to subtract. 1.06s -> 0.39s at 200k cells, 4 hops. - `nhood_aggregate`: one primitive behind all three flavors, bit-identical to the embedders it replaces across 8 configurations. Unexposed. - Embedders are functions, not classes, and the ABC is gone. Dispatch attaches to functions the library owns, not to third-party subclasses (#1151). - Sparse in, sparse out, as upstream does -- the concatenation, which is `hops` times the width of the features, is no longer densified before PCA. - `Array | CSBase` from fast_array_utils instead of `NDArrayA` (#1245). test_niche.py goes 21 -> 46; every fix has a test that fails without it.
This branch has not been deployed
4 participants
Summary
scverse-backends>=0.0.3for Squidpy backend dispatch.rapids-singlecell, withcuda,rapids, andrapids_singlecellaliases.spatial_autocorr,co_occurrence,ligrec, andcalculate_niche.backendusage forspatial_autocorr,co_occurrence, andligrecthrough explicit compatibility handling.scverse_backends.testing.run_conformancewith Squidpy-owned CPU-vs-backend checks.Backend Contract
Backends register an adapter under the
squidpy.backendsentry-point group. The companion RAPIDS SingleCell change supplies that adapter; this host PR keeps the dispatcher configuration and aliases identical to the Scanpy integration.Verification
uv run --group test pytest -q tests/test_backends.pyuv run --group test pytest -q tests/graph/test_ppatterns.pyuv run --group test pytest -q tests/graph/test_ligrec.py -k 'parallel_backend or legacy_parallel_backend'uv run ruff check ...