docs: fix extension-guide review findings, rename phase-one bundle hook

Six findings from a read-through of the new extension guide, plus the API
rename one of them turned into.

The "process local tokens" note sat in the guide index with nothing around it to
explain what a token was or why the reader should care. It moves to a new
`extension_codec_durable_metadata` section in `codecs.md`, where the reader is
already thinking about what goes in a payload: what to encode, then what the
examples do instead, and the three consequences that follow from parking live
objects in a process-local map — no double decode, no fan-out, and a leak for
any plan that never reaches a decoder. The checklist item now points there
instead of at the guide index.

The hook reference loses its `Capsule name` column, which restated
`datafusion_<thing>` for every row when the naming rule already derives it, and
gains a `Contributes` column instead. The argument column stays: those four
values are protocol, not a signature, and only 4 of the 18 hooks have a Python
definition to link at all — the rest are host-side imports, so links into Rust
source would rot faster than the table. Staleness is handled by
`test_hook_reference_table_lists_every_hook` instead, which greps `crates/` and
`python/datafusion/` for `__datafusion_*__` and diffs the set against the table
rows. Verified it fails when a row is dropped.

`capsule-protocol.md` described `abi_stable`, which datafusion-ffi no longer
uses. It now describes `stabby` and the part that is not stabby: `FFI_Option`
and `FFI_Result` are datafusion-ffi's own, because stabby's require
`T: IStable` and the `FFI_*` structs hold self-referential function pointers.
The conversion example converts to `Arc<dyn TableProvider>` rather than naming
`ForeignTableProvider`, since the `From` impl compares library markers and
returns the original `Arc` when both sides are the same library. Three other
snippets on that page had gone stale with it: `FFI_TableProvider::new` with
three arguments, `PyCapsule::new_bound`, and a receiving snippet whose variable
was named `codec`.

In `table-providers.md` all five `Registered with` cells now render
`Receiver.method`, so the schema row reads `Catalog.register_schema` rather
than a bare dotted path, with one sentence on reaching a `Catalog` first.
`Other session components` was in `functions.md`, where an optimizer rule and a
config struct are neither functions nor tables; it becomes its own page,
`other-components.md`, carrying the `extension_other_hooks` label so the index
rows still resolve.

Three guide pages named individual tests, which invites exactly the divergence
the reference is supposed to prevent. They name the suite now.

Finally the phase-one bundle hook. `__datafusion_session_extension__` reused
the name of the whole thing it is a hook on — `with_extensions` takes
extensions and `SessionExtensionExportable` is the bundle protocol — while its
sibling `__datafusion_session_planner__` is named for its content. It is now
`__datafusion_session_components__`, matching both its sibling and the
`SessionExtensionComponents` it returns, which leaves room for the UDF and
provider fields that will join the codec fields later. The protocol class
follows it to `SessionComponentsExportable`, since that file's convention is
one class per hook name. Neither name has shipped, so the upgrade guide needs
no before-and-after; it introduces both hooks as new in this release and names
the new one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: timsaucer

.ai/skills/ffi-capsule-protocol/SKILL.md

@@ -70,7 +70,7 @@ Wrap `fallback` and delegate to it; returning a planner that ignores it discards
 every layer beneath, including one the session already had. It runs after every
 bundle's codecs are installed, so `ctx` carries the final chains.
 
-That is also the only hook where it does. `__datafusion_session_extension__`
+That is also the only hook where it does. `__datafusion_session_components__`
 runs before anything is installed, so its `ctx` still carries the chains the
 receiver had — the same session, and the same task-context provider, but not
 this call's codecs, not even your own. Read the host's codec chains in the

docs/source/conf.py

@@ -123,7 +123,7 @@ def autoapi_skip_member_fn(app, what, name, obj, skip, options) -> bool:  # noqa
         ("class", "datafusion.SessionContext"),
         ("class", "datafusion.QueryPlannerExportable"),
         ("class", "datafusion.SessionExtensionComponents"),
-        ("class", "datafusion.SessionExtensionExportable"),
+        ("class", "datafusion.SessionComponentsExportable"),
         ("class", "datafusion.SessionPlannerExportable"),
         ("module", "datafusion.common"),
         # Duplicate modules (skip module-level docs to avoid duplication)

docs/source/extension-guide/bundles.md

@@ -40,7 +40,7 @@ bundle object implementing one or both of two hooks:
 
 ```python
 class MyEngineExtension:
-    def __datafusion_session_extension__(self, ctx: SessionContext) -> SessionExtensionComponents:
+    def __datafusion_session_components__(self, ctx: SessionContext) -> SessionExtensionComponents:
         # Phase one. Create fresh components bound to `ctx` on every call.
         return SessionExtensionComponents(
             logical_extension_codecs=(self._make_logical_codec(ctx),),
@@ -95,7 +95,7 @@ class Bundle:
     def __init__(self, codec_id):
         self.codec_id = codec_id
 
-    def __datafusion_session_extension__(self, ctx):
+    def __datafusion_session_components__(self, ctx):
         # Fresh components on every call, bound to the `ctx` handed in.
         # Never cache these, and never retain `ctx`.
         return SessionExtensionComponents(
@@ -133,7 +133,7 @@ routes to the codec that wrote the payload. A session holds exactly **one**
 query planner, so planners cannot accumulate — they compose by *nesting*, each
 wrapping the one before it and delegating to it for work it does not handle.
 
-So `with_extensions` runs every `__datafusion_session_extension__` and installs
+So `with_extensions` runs every `__datafusion_session_components__` and installs
 all the codecs, and only then runs each `__datafusion_session_planner__`, in
 argument order, handing each the planner built so far. Two consequences worth
 holding onto:
@@ -204,8 +204,8 @@ class CodecsOf:
     def __init__(self, inner):
         self.inner = inner
 
-    def __datafusion_session_extension__(self, ctx):
-        return self.inner.__datafusion_session_extension__(ctx)
+    def __datafusion_session_components__(self, ctx):
+        return self.inner.__datafusion_session_components__(ctx)
 
 
 class PlannerOf:

docs/source/extension-guide/capsule-protocol.md

@@ -35,12 +35,17 @@ review the code and documentation in the [datafusion-ffi] crate.
 
 Our FFI implementation is narrowly focused on sharing data and functions with
 Rust backed libraries. This allows us to use the
-[abi_stable crate](https://crates.io/crates/abi_stable). This is an excellent
-crate that allows for easy conversion between Rust native types and FFI-safe
-alternatives. For example, if you needed to pass a `Vec<String>` via FFI, you
-can simply convert it to an `RVec<RString>` in an intuitive manner. It also
-supports features like `RResult` and `ROption` that do not have an obvious
-translation to a C equivalent.
+[stabby crate](https://crates.io/crates/stabby), which converts between Rust
+native types and FFI-safe alternatives. For example, if you needed to pass a
+`Vec<String>` via FFI, you can convert it to a
+`stabby::vec::Vec<stabby::string::String>` — the crate's own examples alias
+these as `SVec` and `SString`, which is the convention [datafusion-ffi] follows
+too.
+
+For `Option` and `Result`, [datafusion-ffi] defines its own `FFI_Option<T>` and
+`FFI_Result<T>` rather than using stabby's. Stabby's versions require
+`T: IStable` for niche optimization, and many of the `FFI_*` structs hold
+self-referential function pointers that cannot implement it.
 
 ## `FFI_` on the provider, `Foreign` on the receiver
 
@@ -51,17 +56,19 @@ defined a custom
 and you want to create a sharable FFI counterpart, you could write:
 
 ```rust
-let my_provider = MyTableProvider::default();
-let ffi_provider = FFI_TableProvider::new(Arc::new(my_provider), false, None);
+let my_provider = Arc::new(MyTableProvider::default());
+let ffi_provider = FFI_TableProvider::new_with_ffi_codec(my_provider, false, None, codec);
 ```
 
+where `codec` is the host's logical codec, read off the argument your getter
+was handed — see {ref}`extension_getter_argument`.
+
 If you were interfacing with a library that provided the above
-`FFI_TableProvider` and you needed to turn it back into a `TableProvider`, you
-can turn it into a `ForeignTableProvider`, which implements the `TableProvider`
-trait:
+`FFI_TableProvider` and you needed a usable `TableProvider` back, you convert
+it into an `Arc<dyn TableProvider>`:
 
 ```rust
-let foreign_provider: ForeignTableProvider = ffi_provider.into();
+let provider: Arc<dyn TableProvider> = (&ffi_provider).into();
 ```
 
 If you review the code in [datafusion-ffi] you will find that each of the
@@ -74,6 +81,13 @@ example we're showing, this means the code that has written the underlying
 structures with the `Foreign` prefix are to be used by the receiver. In this
 case, it is the `datafusion-python` library.
 
+Convert to the trait object rather than naming `ForeignTableProvider` yourself.
+The conversion compares the provider's library marker against the receiver's:
+when both sides turn out to be the same shared library it hands back the
+original `Arc` and skips the boundary entirely, and only otherwise wraps it in
+a `ForeignTableProvider`. Which one you get is an implementation detail, and
+both implement `TableProvider`.
+
 ## Wrapping it in a capsule
 
 In order to share these FFI structures, we need to wrap them in some kind of
@@ -82,20 +96,20 @@ described in {ref}`extension_why_ffi`, we use `PyCapsule`. We can create a
 `PyCapsule` for our provider thusly:
 
 ```rust
-let name = CString::new("datafusion_table_provider")?;
-let my_capsule = PyCapsule::new_bound(py, provider, Some(name))?;
+PyCapsule::new_with_value(py, ffi_provider, cr"datafusion_table_provider")
 ```
 
-On the receiving side, turn this pycapsule object into the
-`FFI_TableProvider`, which can then be turned into a `ForeignTableProvider`;
-the associated code is:
+On the receiving side, read the `FFI_TableProvider` back out of the capsule and
+convert it, which is what `table_provider_from_pycapsule` in `crates/util` does:
 
 ```rust
-let capsule = capsule.cast::<PyCapsule>()?;
+validate_pycapsule(capsule, "datafusion_table_provider")?;
 let data: NonNull<FFI_TableProvider> = capsule
-    .pointer_checked(Some(name))?
+    .pointer_checked(Some(c"datafusion_table_provider"))?
     .cast();
-let codec = unsafe { data.as_ref() };
+let ffi_provider = unsafe { data.as_ref() };
+check_ffi_version("table provider", unsafe { (ffi_provider.version)() })?;
+let provider: Arc<dyn TableProvider> = ffi_provider.into();
 ```
 
 ## The naming rule
@@ -112,7 +126,7 @@ must return a capsule named `datafusion_table_provider`. Return a capsule with
 the wrong name and the import fails with an error naming both the name found
 and the name expected, rather than reading the pointer as the wrong type.
 
-The full list of hooks and their capsule names is in the
+The full list of hooks is in the
 {ref}`hook reference <extension_guide>`. `TableProvider` was the first
 extension written this way and is the most thoroughly implemented; every hook
 added since follows the same pattern.
@@ -196,7 +210,7 @@ getter and `__datafusion_codec_id__` — everything the protocol asks of it —
 `isinstance(session, SessionContext)` is `False` in Python even though its
 `repr` reads `datafusion.SessionContext`.
 
-The two bundle hooks are the exception: `__datafusion_session_extension__` and
+The two bundle hooks are the exception: `__datafusion_session_components__` and
 `__datafusion_session_planner__` are dispatched from Python by
 {py:meth}`~datafusion.SessionContext.with_extensions`, so they receive the
 wrapper. See {doc}`bundles`.

docs/source/extension-guide/checklist.md

@@ -93,7 +93,7 @@ publish. Each links to the page that explains it.
 - [ ] **Your production codec serializes durable metadata**, not a
       process-local token. The examples in this repository use tokens to make
       ownership observable; that is a demonstration, not a pattern.
-      → {ref}`extension_guide`
+      → {ref}`extension_codec_durable_metadata`
 - [ ] **You have integration tests across a real FFI boundary.** The two
       example crates in this repository are the pattern: build the cdylib,
       install the wheel, then exercise it from Python.

docs/source/extension-guide/codecs.md

@@ -50,6 +50,24 @@ A codec that also ships to hosts which dispatch differently may still want its
 own guard against foreign payloads. Keeping one is fine; it is simply not
 needed for the datafusion-python path.
 
+(extension_codec_durable_metadata)=
+
+## Encode metadata, not a handle to a live object
+
+Your payload has to be enough to rebuild the object somewhere your process is
+not. Write the metadata a fresh instance can be constructed from — a path, a
+connection string, a schema, the options the object was created with.
+
+The example codecs in this repository do not do this, and it is worth knowing
+before copying them. They keep a process-local `HashMap` of live providers and
+encode an integer token into it: encoding inserts, decoding removes. That makes
+Rust type identity observable across three separately loaded libraries in one
+test, which is what the examples exist to show. It also means a decode consumes
+its token, so the same bytes cannot be decoded twice, one encoded plan cannot
+fan out to several readers, and a plan that never reaches a decoder keeps its
+provider alive for the life of the process. A real codec has none of those
+properties because it does not park the object anywhere.
+
 (extension_codec_ids)=
 
 ## Codec ids
@@ -133,8 +151,7 @@ after it. The query still succeeds. What changes is which library wrote the
 bytes — so a plan that has to decode in another process now needs whichever
 library happened to win, not the one whose node it is.
 `MyPhysicalExtensionCodec` in [`datafusion-ffi-example`] claims this way, and
-`test_a_greedy_codec_installed_first_claims_another_librarys_node` pins the
-consequence.
+the query-planner example's test suite pins the consequence.
 
 Two rules of thumb:
 

docs/source/extension-guide/functions.md

@@ -103,51 +103,4 @@ on your decoder — with an empty payload there is no id to route on, so your
 
 `NameOnlyUdfCodec` in [`datafusion-ffi-example`] is the worked case.
 
-(extension_other_hooks)=
-
-## Other session components
-
-Two further hooks contribute things that are neither data nor functions. Both
-take no argument and both are implemented in [`datafusion-ffi-example`].
-
-**`__datafusion_physical_optimizer_rule__`** contributes a rule that rewrites
-physical plans, installed with
-{py:meth}`~datafusion.SessionContext.add_physical_optimizer_rule`. Reach for
-this rather than a {doc}`query planner <query-planners>` when you want to
-adjust the plan DataFusion produced rather than produce it yourself — it is
-much the smaller commitment, and rules accumulate where planners nest.
-
-```rust
-fn __datafusion_physical_optimizer_rule__<'py>(
-    &self,
-    py: Python<'py>,
-) -> PyResult<Bound<'py, PyCapsule>> {
-    let rule: Arc<dyn PhysicalOptimizerRule + Send + Sync> = Arc::new(self.clone());
-    let runtime = get_tokio_runtime().handle().clone();
-    let ffi = FFI_PhysicalOptimizerRule::new(rule, Some(runtime));
-
-    PyCapsule::new_with_value(py, ffi, cr"datafusion_physical_optimizer_rule")
-}
-```
-
-**`__datafusion_extension_options__`** contributes typed configuration entries
-that your components can read back out of the session config, installed with
-{py:meth}`SessionConfig.with_extension <datafusion.SessionConfig.with_extension>`.
-`FFI_ExtensionOptions` carries no version field, so it is one of the three
-components that cannot be version-checked on import.
-
-```rust
-fn __datafusion_extension_options__<'py>(
-    &self,
-    py: Python<'py>,
-) -> PyResult<Bound<'py, PyCapsule>> {
-    let mut config = FFI_ExtensionOptions::default();
-    config
-        .add_config(self)
-        .map_err(|e| PyRuntimeError::new_err(e.to_string()))?;
-
-    PyCapsule::new_with_value(py, config, cr"datafusion_extension_options")
-}
-```
-
 [`datafusion-ffi-example`]: https://github.com/apache/datafusion-python/tree/main/examples/datafusion-ffi-example

docs/source/extension-guide/index.md

@@ -69,11 +69,6 @@ The session owns the codecs used for the exchange and supplies them to the
 foreign planner. That is what lets the planner decode provider-owned objects,
 and lets datafusion-python decode the physical plan the planner returns.
 
-:::{note}
-The example codecs use process-local tokens to demonstrate ownership.
-A production codec should serialize durable metadata instead.
-:::
-
 ## Hook reference
 
 Every integration point is a dunder method named `__datafusion_*__`. The
@@ -83,36 +78,45 @@ FFI-safe struct. See {doc}`capsule-protocol` for what that means and
 {ref}`extension_getter_argument` for the argument every getter in the middle
 group receives.
 
-| Hook | Capsule name | Argument | Documented on |
+Each hook's capsule name follows from its own name by
+{ref}`the naming rule <extension_capsule_protocol>`, so it is not repeated
+here.
+
+| Hook | Contributes | Argument | Documented on |
 | --- | --- | --- | --- |
-| `__datafusion_table_provider__` | `datafusion_table_provider` | codec source | {doc}`table-providers` |
-| `__datafusion_table_provider_factory__` | `datafusion_table_provider_factory` | codec source | {doc}`table-providers` |
-| `__datafusion_catalog_provider__` | `datafusion_catalog_provider` | codec source | {doc}`table-providers` |
-| `__datafusion_catalog_provider_list__` | `datafusion_catalog_provider_list` | codec source | {doc}`table-providers` |
-| `__datafusion_schema_provider__` | `datafusion_schema_provider` | codec source | {doc}`table-providers` |
-| `__datafusion_table_function__` | `datafusion_table_function` | session | {doc}`functions` |
-| `__datafusion_scalar_udf__` | `datafusion_scalar_udf` | none | {doc}`functions` |
-| `__datafusion_aggregate_udf__` | `datafusion_aggregate_udf` | none | {doc}`functions` |
-| `__datafusion_window_udf__` | `datafusion_window_udf` | none | {doc}`functions` |
-| `__datafusion_logical_extension_codec__` | `datafusion_logical_extension_codec` | session | {doc}`codecs` |
-| `__datafusion_physical_extension_codec__` | `datafusion_physical_extension_codec` | session | {doc}`codecs` |
-| `__datafusion_codec_id__` | *not a capsule — a string attribute* | — | {doc}`codecs` |
-| `__datafusion_query_planner__` | `datafusion_query_planner` | session | {doc}`query-planners` |
-| `__datafusion_session_extension__` | *not a capsule — returns components* | `ctx` | {doc}`bundles` |
-| `__datafusion_session_planner__` | `datafusion_query_planner`, or `None` | `ctx`, `fallback` | {doc}`bundles` |
-| `__datafusion_physical_optimizer_rule__` | `datafusion_physical_optimizer_rule` | none | {ref}`extension_other_hooks` |
-| `__datafusion_extension_options__` | `datafusion_extension_options` | none | {ref}`extension_other_hooks` |
-| `__datafusion_task_context_provider__` | `datafusion_task_context_provider` | none | {ref}`extension_task_context_provider` |
-
-Two rows are not like the others. `__datafusion_task_context_provider__` is
+| `__datafusion_table_provider__` | one table | codec source | {doc}`table-providers` |
+| `__datafusion_table_provider_factory__` | tables built by `CREATE EXTERNAL TABLE` | codec source | {doc}`table-providers` |
+| `__datafusion_catalog_provider__` | a named set of schemas | codec source | {doc}`table-providers` |
+| `__datafusion_catalog_provider_list__` | the whole catalog namespace | codec source | {doc}`table-providers` |
+| `__datafusion_schema_provider__` | a named set of tables | codec source | {doc}`table-providers` |
+| `__datafusion_table_function__` | a table-valued function | session | {doc}`functions` |
+| `__datafusion_scalar_udf__` | a scalar function | none | {doc}`functions` |
+| `__datafusion_aggregate_udf__` | an aggregate function | none | {doc}`functions` |
+| `__datafusion_window_udf__` | a window function | none | {doc}`functions` |
+| `__datafusion_logical_extension_codec__` | a logical codec | session | {doc}`codecs` |
+| `__datafusion_physical_extension_codec__` | a physical codec | session | {doc}`codecs` |
+| `__datafusion_codec_id__` | the wire id a codec's payloads carry | — | {ref}`extension_codec_ids` |
+| `__datafusion_query_planner__` | a query planner | session | {doc}`query-planners` |
+| `__datafusion_session_components__` | a bundle's codecs | `ctx` | {doc}`bundles` |
+| `__datafusion_session_planner__` | a bundle's planner, wrapping `fallback` | `ctx`, `fallback` | {doc}`bundles` |
+| `__datafusion_physical_optimizer_rule__` | a physical optimizer rule | none | {ref}`extension_other_hooks` |
+| `__datafusion_extension_options__` | typed entries in the session config | none | {ref}`extension_other_hooks` |
+| `__datafusion_task_context_provider__` | the host's task context | none | {ref}`extension_task_context_provider` |
+
+Three rows are not like the others. `__datafusion_task_context_provider__` is
 implemented by the **host**, not by your library — you read it off the session
-you are handed. And `__datafusion_codec_id__` is a plain string attribute
-rather than a method returning a capsule.
+you are handed. `__datafusion_codec_id__` is a plain string attribute rather
+than a method returning a capsule. And the two `session_` hooks are dispatched
+from Python and return objects rather than capsules.
 
 "codec source" in the argument column means the value is something you can
 read the host's logical extension codec off, which is not always a session.
 {ref}`extension_getter_argument` explains why, and what to do with it.
 
+`python/tests/test_docstrings.py` compares this table against the hook names
+the package actually dispatches, so a hook added or renamed without touching
+this page fails the suite.
+
 ```{toctree}
 :maxdepth: 2
 
@@ -123,6 +127,7 @@ functions
 codecs
 bundles
 query-planners
+other-components
 sessions
 checklist
 ```