Skip to content

Run examples/*.py in CI, fix the broken example, and give silent ones output - #1736

Open
voidstackloop wants to merge 1 commit into
apache:mainfrom
voidstackloop:examples-ci
Open

Run examples/*.py in CI, fix the broken example, and give silent ones output#1736
voidstackloop wants to merge 1 commit into
apache:mainfrom
voidstackloop:examples-ci

Conversation

@voidstackloop

Copy link
Copy Markdown

Which issue does this PR close?

Closes #1728.

Rationale for this change

Nothing in CI runs the top-level examples/*.py scripts, so they silently drift
from the API and from each other. Two consequences are already in the tree:

  • examples/csv-read-options.py crashes: it reads data.csv and
    data.csv.gz, neither of which exists in the repository.
  • Nine examples end in assert with no output, so a reader cannot tell a
    working example from a script that does nothing.

Adding a CI job is what keeps this from recurring.

What changes are included in this PR?

  • examples/csv-read-options.py now writes its own small CSV and gzipped CSV
    into a temporary directory, so it is self-contained and runs without external
    data.
  • The nine silent examples (export.py, import.py, python-udaf.py,
    python-udf.py, query-pyarrow-data.py, sql-to-pandas.py,
    sql-using-python-udaf.py, sql-using-python-udf.py, substrait.py) now
    print their result, keeping their existing assertions.
  • .github/workflows/test.yml gains a Run example scripts step (Python 3.12
    entry) that runs every examples/*.py against the built wheel. It installs
    the extra runtime dependencies some examples use (pandas, polars,
    matplotlib) and skips an explicit list of examples that cannot run in CI:
    • sql-parquet-s3.py — needs network access and AWS credentials
    • sql-parquet.py, dataframe-parquet.py, sql-to-pandas.py — need the NYC
      taxi parquet file documented in examples/README.md
    • python-udf-comparisons.py — needs the TPC-H dataset generated later in the
      workflow
    • ray_pickle_expr.py — needs the optional (heavy) ray dependency

Are there any user-facing changes?

No API changes. The examples are documentation; they are now runnable, print
their results, and CI will catch it if they break again.

Verification

  • Ran every non-skipped example against a datafusion==54.0.0 wheel in a fresh
    venv: all pass. multiprocessing_pickle_expr.py and ray_pickle_expr.py were
    not run locally (Windows has no forkserver and Ray is unavailable there);
    they run on the Linux CI.
  • ruff check / ruff format --check (pinned 0.15.1), codespell and
    actionlint (v1.7.12) pass on the changed files.

… output

Nothing in CI ran the top-level examples/*.py scripts, so they drifted out of
sync with the API. examples/csv-read-options.py crashed because it reads
data.csv and data.csv.gz that do not exist in the repository, and nine other
examples end in asserts without printing anything, so a reader cannot tell a
working example from a no-op.

- Make csv-read-options.py self-contained: it now writes its own small CSV and
  gzipped CSV into a temporary directory.
- Add a terminal call to the examples that printed nothing so each one shows
  its result.
- Add a CI step, gated to the 3.12 abi3 entry, that runs every examples/*.py
  script against the built wheel with an explicit skip list for examples that
  need network/credentials, hand-downloaded data, generated TPC-H data, or the
  optional Ray dependency.

Closes apache#1728
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nothing in CI runs examples/*.py, and one of them is broken

1 participant