Add a worked example of three extension libraries in one distributed query - #1721

Open

timsaucer wants to merge 31 commits into

mainfrom

feat/distributed-extensions-example

Commits on Sep 14, 2026

Add dfx_storage: a codec that writes durable metadata

timsaucer
and
claude
committed
34667a4
Add dfx_udfs: the library that cannot be installed as a bundle

timsaucer
and
claude
committed
7b78f27
Add dfx_engine: a toy distributed engine in both halves

timsaucer
and
claude
committed
1a6fbd8
Add the queries and the cross-library integration tests

timsaucer
and
claude
committed
4f0fbef
Wire the distributed example into CI, and say which example is which

timsaucer
and
claude
committed
7f5ac79
Document what a worker has to reproduce, and three findings that cost time

timsaucer
and
claude
committed
76d9aca
Build the example libraries on the host, and prime their deps once

timsaucer
and
claude
committed
fc14b14
Number the stages, and ship all of them

timsaucer
and
claude
committed
d0e50e2
Read the third codec id instead of copying it

timsaucer
and
claude
committed
f911e1f
Make the example's run instructions work

timsaucer
and
claude
committed
3a5736c
Cover build_session's own consistency check

timsaucer
and
claude
committed
1fc5207
Point local_session.rs at a section that exists

timsaucer
and
claude
committed
4a17b5d
Say why table_options_mut panics on purpose

timsaucer
and
claude
committed
4f0ae01
Link the two sections this PR added

timsaucer
and
claude
committed
3035109
Decode a provider with the schema from the plan

timsaucer
and
claude
committed
502780d
Refuse a malformed projection instead of dropping it

timsaucer
and
claude
committed
bdc9bae
Raise Execution, not Internal, for a bad payload

timsaucer
and
claude
committed
1fbda34
Stop converting a fallback this planner never calls

timsaucer
and
claude
committed
e5693df
Clear the small stuff off the distributed example

timsaucer
and
claude
committed
f50a1b7
Refuse a shuffle directory that already holds stage output

timsaucer
and
claude
committed
5fd7907
Check a shuffle file's schema instead of adopting it

timsaucer
and
claude
committed
c4629d5
Size a scalar function's output from the batch, not its arguments

timsaucer
and
claude
committed
47700f9
Say what the example-test gate actually does

timsaucer
and
claude
committed
0a3445b
Keep the driver's scratch out of the results directory

timsaucer
and
claude
committed
334643c
Write workflow comments so counts and versions cannot rot

timsaucer
and
claude
committed
e26520d
Name the codecs that actually park live objects

timsaucer
and
claude
committed
d25e184
Fix what the new docstrings claim, and where they claim it

timsaucer
and
claude
committed
93ecace
Reap worker processes on every driver exit path

timsaucer
and
claude
committed
58a52a6
Make the tests fail for the implementations they claim to catch

timsaucer
and
claude
committed
3c410c5
Close the review's smaller findings

timsaucer
and
claude
committed
9c16b97
Let tpchgen-cli shard the data instead of resharding it
timsaucer
committed
93c9353

Navigation Menu

Uh oh!

Add a worked example of three extension libraries in one distributed query - #1721

Add a worked example of three extension libraries in one distributed query#1721
timsaucer wants to merge 31 commits into
mainfrom
feat/distributed-extensions-example

Commits on Sep 14, 2026

Add dfx_storage: a codec that writes durable metadata

Add dfx_udfs: the library that cannot be installed as a bundle

Add dfx_engine: a toy distributed engine in both halves

Add the queries and the cross-library integration tests

Wire the distributed example into CI, and say which example is which

Document what a worker has to reproduce, and three findings that cost time

Build the example libraries on the host, and prime their deps once

Number the stages, and ship all of them

Read the third codec id instead of copying it

Make the example's run instructions work

Cover build_session's own consistency check

Point local_session.rs at a section that exists

Say why table_options_mut panics on purpose

Link the two sections this PR added

Decode a provider with the schema from the plan

Refuse a malformed projection instead of dropping it

Raise Execution, not Internal, for a bad payload

Stop converting a fallback this planner never calls

Clear the small stuff off the distributed example

Refuse a shuffle directory that already holds stage output

Check a shuffle file's schema instead of adopting it

Size a scalar function's output from the batch, not its arguments

Say what the example-test gate actually does

Keep the driver's scratch out of the results directory

Write workflow comments so counts and versions cannot rot

Name the codecs that actually park live objects

Fix what the new docstrings claim, and where they claim it

Reap worker processes on every driver exit path

Make the tests fail for the implementations they claim to catch

Close the review's smaller findings

Let tpchgen-cli shard the data instead of resharding it