Skip to content

AGENTS.md

Latest commit

 

History

History
187 lines (144 loc) · 8.71 KB

AGENTS.md

File metadata and controls

187 lines (144 loc) · 8.71 KB
Raw

Agent Instructions for Contributors

This file is for agents working on the datafusion-python project (developing, testing, reviewing). If you need to use the DataFusion DataFrame API (write queries, build expressions, understand available functions), see the user-facing skill at SKILL.md.

Skills

This project uses AI agent skills stored in .ai/skills/. Each skill is a directory containing a SKILL.md file with instructions for performing a specific task.

Skills follow the Agent Skills open standard. Each skill directory contains:

  • SKILL.md — The skill definition with YAML frontmatter (name, description, argument-hint) and detailed instructions.
  • Additional supporting files as needed.

To discover what skills are available, list .ai/skills/ and read each SKILL.md. The frontmatter name and description fields summarize the skill's purpose. Some descriptions begin with TRIGGER —; those are not tasks to run on request but conventions to read before writing code that meets the stated condition.

FFI Capsule Protocol

The __datafusion_*__ capsule getters are one protocol with a settled convention. Before adding or changing one, read .ai/skills/ffi-capsule-protocol/SKILL.md.

Documentation Sources

Search and edit docs/source/. docs/temp/ is gitignored build output that grep -r will surface with stale copies of the same pages.

Before changing a public API, check docs/source/user-guide/upgrade-guides.md for how the same API family was migrated previously. Follow the established pattern rather than inventing one.

Pull Requests

Every pull request must follow the template in .github/pull_request_template.md. The description must include these sections:

  1. Which issue does this PR close? — Link the issue with Closes #NNN.
  2. Rationale for this change — Why the change is needed (skip if the issue already explains it clearly).
  3. What changes are included in this PR? — Summarize the individual changes.
  4. Are there any user-facing changes? — Note any changes visible to users (new APIs, changed behavior, new files shipped in the package, etc.). If there are breaking changes to public APIs, add the api change label and add a section to docs/source/user-guide/upgrade-guides.md showing the before and after. This applies to FFI hook method signatures, which extension libraries implement.

Pre-commit Checks

Always run pre-commit checks before committing. The hooks are defined in .pre-commit-config.yaml and run automatically on git commit if pre-commit is installed as a git hook. To run all hooks manually:

pre-commit run --all-files

Fix any failures before committing.

Test Coverage

Always prefer Python coverage — a doctest example in a docstring, or a pytest case. The user-facing Python surface is the first line of defense and the primary focus, so behavior should be pinned where users actually meet it.

CI does not run Rust tests. No workflow invokes cargo test; the only Rust checks are cargo fmt --check and cargo clippy --no-deps --all-targets. --all-targets compiles #[cfg(test)] code, so a Rust test cannot rot into a non-compiling state, but it is never executed and a behavioral regression will not fail the build. A Rust test added today is dead weight.

Adding a cargo test job is not a one-line change: crates/core/Cargo.toml enables pyo3/extension-module unconditionally, so the test binary fails to link against Py_* symbols on Linux. The feature would have to be gated first.

Write a Rust test only when the behavior is genuinely unreachable from Python, and wire up CI in the same change so it actually runs. Before concluding it is unreachable, check the suites that already exist:

  • python/tests/ — the main suite. Run pytest python/, not pytest python/tests/: --doctest-modules is on by default and the narrower path skips the doctests in python/datafusion/.
  • examples/datafusion-ffi-example/python/tests/ and examples/datafusion-ffi-query-planner-example/python/tests/ — integration coverage across a real FFI boundary, for anything involving extension codecs, table providers, query planners, or capsule export. These need the example crates built (maturin build, then install the wheel).
  • examples/tpch/ — end-to-end query coverage.

Prefer asserting observable behavior over internal accessors. A test that checks a getter can pass while the path a user actually takes is broken.

Python Function Docstrings

Every Python function must include a docstring with usage examples.

  • Examples are required: Each function needs at least one doctest-style example demonstrating basic usage.
  • Optional parameters: If a function has optional parameters, include separate examples that show usage both without and with the optional arguments. Pass optional arguments using their keyword name (e.g., step=dfn.lit(3)) so readers can immediately see which parameter is being demonstrated.
  • Reuse input data: Use the same input data across examples wherever possible. The examples should demonstrate how different optional arguments change the output for the same input, making the effect of each option easy to understand.
  • Alias functions: Functions that are simple aliases (e.g., list_sort aliasing array_sort) only need a one-line description and a See Also reference to the primary function. They do not need their own examples.

One canonical home per claim

A claim lives where the reader already is when they need it — exactly once.

  • A property of one callable's arguments, return value, or errors → that callable's docstring.
  • A property of a type that several callables share → the class docstring. (Session sharing and context lifetime live on SessionContext, not on each of the five with_* methods.)
  • Why the API is shaped this way, a multi-library recipe, Rust-side code, a trade-off, or a limitation with an upstream issue → the narrative guide under docs/source/.

Everywhere else: one sentence plus one Sphinx role. A docstring may state a claim the guide also makes; it must not argue it — no "because", no "the reason is", no counter-argument.

The test: if a paragraph would survive being moved into the guide unchanged, move it. python/tests/test_docstrings.py enforces a length ceiling, which is the symptom this rule treats.

When you point at the guide, use a real :ref: to a specific label. Prose saying "see the extensions guide" with no role is a dead end in the rendered HTML.

Examples that need a compiled extension

Some APIs cannot be demonstrated without a built FFI extension library, which this package does not ship. The convention is:

  1. A runnable example block first, using only the wheel. A SessionContext satisfies the capsule-getter protocols, so its own exported capsule stands in for a real library's in a doctest.
  2. Then a # doctest: +SKIP block showing real usage, at most a few lines, with one line of prose naming the test that runs it for real.
  3. Every +SKIP block needs that mirror. See test_with_extensions_docstring_example_still_runs in examples/datafusion-ffi-query-planner-example, which parses the live docstring, drops the skip, and executes it — so a renamed method or a wrong expected output fails there.

Aggregate and Window Function Documentation

When adding or updating an aggregate or window function, ensure the corresponding site documentation is kept in sync:

  • Aggregations: docs/source/user-guide/common-operations/aggregations.md — add new aggregate functions to the "Aggregate Functions" list and include usage examples if appropriate.
  • Window functions: docs/source/user-guide/common-operations/windows.md — add new window functions to the "Available Functions" list and include usage examples if appropriate.