Skip to content

Restore dynamic Cython modules from shared extension paths - #622

Open
Aschenbath wants to merge 1 commit into
cloudpipe:masterfrom
Aschenbath:support-shared-dynamic-cython
Open

Aschenbath wants to merge 1 commit into
cloudpipe:masterfrom
Aschenbath:support-shared-dynamic-cython

Conversation

@Aschenbath

Copy link
Copy Markdown

Summary

Cython functions and classes produced by pyximport or IPython %%cython can be pickled by module name but fail to load in a fresh worker because the compiled module is outside its import path.

Add the opt-in assume_shared_dynamic_cython_module_folder keyword to Pickler, dump, and dumps. When enabled, named objects from dynamically compiled extensions can be restored from the same absolute extension path on each worker. Normal importable extensions and modules registered for pickling by value retain their existing behavior.

Restore shared extensions through standard import initialization and module locking. Repeated loads retain module identity; conflicting paths fail without replacing the loaded module. The exact-name finder is removed after both successful and failed imports. Document compatibility requirements and add real pyximport/%%cython tests, with Cython as a development dependency.

Fixes #502.

Validation

  • Full suite on Linux CPython 3.12.12: 299 passed, 10 skipped, 2 xfailed. Two additional parametrized reference-behavior controls were added afterward; the final Cython test module passes 35 tests. The full suite was not rerun after those test-only additions.
  • All configured hooks pass for the five changed files. The repository-wide hook run found a Black formatting failure in unchanged tests/cloudpickle_ipykernel_test.py, also reproduced on the base revision; that file is unchanged here.
  • Built wheel and sdist. In a separate virtual environment containing the installed wheel and no Cython, producer and fresh consumer processes invoked with python -I passed real-extension round trips through all three public APIs, including function/class/instance identity.
  • A separate Codex agent independently tested real Cython initialization races, parent-package re-exports, path conflicts, module identity, finder cleanup, and the pure-Python Pickler branch. The final remote files were checked against the reviewed source.
  • Existing fork workflows were left unchanged. A push to the fork's master registered the workflows, but GitHub currently reports zero runs/check runs/statuses for this commit; remote CI has not passed.

Scope and contribution disclosure

Compiled files must exist at the same absolute paths with compatible Python, ABI, platform, and native dependencies. This does not transfer or rebuild native code, support local Cython closures, or snapshot modified native module globals. Real PyPy was not run; the Python pickler branch was exercised on CPython, which does not establish PyPy/cpyext compatibility.

This implementation, tests, and PR text were prepared using OpenAI Codex. Review was performed by a separate Codex agent; no human review is claimed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: support for pickling/unpickling interactively defined Cython functions and classes

1 participant