Support zarr v3

[alejoe91]

Fixes #4014 - UPDATED 16/10/2026

Implement bulk changes from zarr v2 to zarr v3

• create_dataset -> create_array
• Serialization of non JSON serializable objects (recording/sorting/extensions) not supported anymore
• Compressors use zarr.codecs, which wrap numcodecs compressor (e.g., Blosc -> BloscCodec)
• Codec Pipeline: improved generation of codec pipeline to automatically chain filters (ArrayArrayCodec), serializers (standard unless an ArrayBytesCodec is used - e.g. Wavpack), and compressors (BytesBytesCodec)
• Sharding: new sharding option in the zarr save functions (save(..., format="zarr")), with tests on chunks and shards
• Backward compatibility: loading objects written with zarr v2 (recordings, sortings, templates, SortingAnalyzer with extensions), including zarr_class_info, channel_ids dtype handling, and storage_options
• NWB / zarr:
  • zarr links are read from _LINKS for pynwb
  • zarr children iteration is wrapped to skip arrays whose metadata zarr v3 can't parse
  • streaming zarr (stream_mode="zarr") now works with zarr>=3 and its test is re-enabled
  • NWB-zarr tests only run the hdf5 backend when hdmf-zarr is not installed
• Python: requires-python >= 3.11. The hdmf-zarr>=0.14.0 extras are restricted to python_version>='3.12', since hdmf-zarr 0.14.0 requires Python >=3.12. NWB-zarr (pynwb) is therefore unavailable on 3.11, while plain zarr reading/writing and zarr streaming work everywhere.
• Removed the requires_zarr_write test marker (zarr writing is always available) and cleaned up tests
• CI:
  • cross-version serialization workflow checks v2 -> v3 compatibility (including dev) on all supported versions
  • numpy pinned <2.5 in the cross-serialization tests
  • deepinterpolation action disabled
• Docs: updated the compression section

Depends on:

• ProbeInterface: Support zarr v3 probeinterface#387
• NWB (hdmf-zarr) 0.14.0

[alejoe91]

@h-mayorquin not this is Python 3.11, with optional install of hdmf-zarr. Good to merge on my end!

[h-mayorquin]

I am reviewing this in more detail, as mentioned. I found two bugs:

• get_preprocessing_list_from_analyzer fails on the backward compatibility path: it raises for any zarr analyzer written with a previous version.
• A recording that is not JSON-serializable is no longer saved in a zarr analyzer (main falls back to pickle), so the analyzer reloads without its recording and cannot compute the extensions that need it. This includes any recording that went through correct_motion or silence_periods, among others.

As I see it, the PR has the following features:

1. Dropping Python 3.10 (#4873).
2. Reading zarr v2 NWB files with zarr-python 3 (#4875).
3. A cross-version serialization fixture for the SortingAnalyzer (#4876).
4. Changing the SortingAnalyzer zarr format so it no longer uses object arrays: JSON goes into attributes instead of JSON or pickle object arrays, and scikit-learn models are saved as arrays and attributes instead of pickles.
5. Writing with zarr-python 3 (create_array and the v3 codec pipeline).
6. Backward compatibility: reading zarr folders saved with zarr-python 2 (SortingAnalyzer, recordings and sortings) when zarr-python 3 is installed.
7. Sharding.

I think the PR is a little too large, which is why I peeled off 1, 2 and 3, so we can give the zarr-exclusive changes (5, 6 and 7) another concentrated read. I think 4 should be its own change as well, since it is a big change to the on-disk format. The second bug comes from it: the new format has no place for pickled data, and deciding where that data goes is a format decision that deserves its own PR. A focused review of the zarr part will make it easier to find bugs like the ones above, and it is good PR hygiene overall.

Besides that, two concrete problems:

1. Dead code: zarr.config.set({"default_zarr_version": 3}) in zarrextractors.py does nothing. The zarr-python 3 key is default_zarr_format, and it is already 3 by default.
2. core-test.yml is added back, but it was removed in #4822.

Note that I tried to do the peel-offs cleanly so there are no merge conflicts: the files in #4876 are identical to yours. #4875 is the exception, because the NWB part had to be different. zarr-python 3 cannot parse some of the arrays hdmf-zarr wrote in zarr v2 files, so the approach here skips them instead of reading them. hdmf-zarr 0.14 already reads them with NWBZarrV2IO, so #4875 sends v2 files to pynwb and keeps our in-house reader on v3 only.

If you prefer to merge this first, as you asked in #4875, we can merge only the zarr core (5, 6 and 7) here and land the rest afterwards as their own PRs (#4875, #4876 and 4). We can handle that the same way as the Python 3.14 support (#4173): whatever depends on the rest raises a clear "not supported yet" error with zarr-python 3 until it lands. Unlike #4173, that would affect every Python version, not only 3.14, so the rest should land before the next release. That should be easy to decouple and/or solve conflicts with an LLM if you want an expedited merge of this as the core looks good to me once the first bug and the two problems above are fixed.