Skip to content

GH-54903: [C++][OSS-Fuzz fuzzing] Update csv fuzzer and corpora/dict preparation - #54912

Open
arthurscchan wants to merge 3 commits into
apache:mainfrom
arthurscchan:fuzz/csv-options
Open

arthurscchan wants to merge 3 commits into
apache:mainfrom
arthurscchan:fuzz/csv-options

Conversation

@arthurscchan

@arthurscchan arthurscchan commented Oct 21, 2026 •

Copy link
Copy Markdown

Rationale for this change

This PR improves the fuzzing coverage and effectiveness of the Arrow C++ CSV reader, as part of the broader effort described in GH-54903. Currently, the CSV fuzz target reads every input with a fixed set of hardcoded options, and its seed corpus only covers a single dialect, so most of the reader's option space (delimiters, quoting/escaping, null handling, block sizes, dictionary encoding, timestamp parsers, ...) is explored mostly by chance.

This change is coordinated with the OSS-Fuzz side in google/oss-fuzz#16332.

What changes are included in this PR?

  • cpp/src/arrow/csv/fuzz.cc:
    • The fuzz input is now prefixed with a packed, magic-number-prefixed header encoding a set of reader options, following the approach of the Parquet encoding fuzzer. Inputs that do not start with the expected magic are treated as legacy and read with the historical hardcoded options, so existing corpus entries keep working.
    • Each read strategy (single-threaded TableReader, multi-threaded TableReader, StreamingReader, and CountRowsAsync) is given a fresh input stream, since a reader consumes the stream it is passed.
    • When all three table reads succeed, the resulting tables are now checked for equality (differential fuzzing), aborting on inconsistency like the Parquet encoding fuzzer does.
  • cpp/src/arrow/csv/fuzz_header_internal.h (new): the packed FuzzCsvHeader structure (bit-field option groups with static_asserts on the layout), and its serialization and parsing/validation helpers.
  • cpp/src/arrow/csv/generate_fuzz_corpus.cc: generated seed files are now prefixed with the serialized fuzz header matching the dialect used for writing, and several dialect variants are generated (alternate delimiters, quote-all, headerless with autogenerated column names, N/A nulls). The batch-size variant was dropped, as it only exercised the writer's internal chunking, which is irrelevant for the reader.
  • cpp/build-support/fuzzing/pack_corpus.py: files with duplicate content now emit a warning and are skipped instead of failing the corpus packing (some Pandas test CSVs are exact duplicates of each other).
  • cpp/build-support/fuzzing/generate_corpuses.sh: removed the now-unneeded deduplication workaround; documented how dictionaries are used (they are hints to the fuzzing engine, not restrictions) - also documented in cpp/build-support/fuzzing/dicts/arrow-csv-fuzz.dict.

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

Code and description were drafted with the help of an AI coding agent, then reviewed, revised and validated by the author before submission.

Signed-off-by: Arthur Chan <arthur.chan@adalogics.com>
@arthurscchan arthurscchan changed the title OSS-Fuzz: Update csv fuzzer and corpora/dict preparation GH-54903: [OSS-Fuzz fuzzing] Update csv fuzzer and corpora/dict preparation Oct 21, 2026
@arthurscchan arthurscchan changed the title GH-54903: [OSS-Fuzz fuzzing] Update csv fuzzer and corpora/dict preparation GH-54903: [C++][OSS-Fuzz fuzzing] Update csv fuzzer and corpora/dict preparation Oct 22, 2026
@pitrou

pitrou commented Oct 22, 2026

Copy link
Copy Markdown
Member

Thanks for the PR @arthurscchan . Is there any reason this doesn't reuse our standard PR description template?

@pitrou pitrou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for doing this. On the principle this is interesting, but I think there are number of problems with the current implementation.

Comment thread cpp/src/arrow/csv/fuzz.cc Resolved
Comment thread cpp/src/arrow/csv/fuzz.cc Resolved
Comment thread cpp/src/arrow/csv/fuzz.cc Resolved
Comment thread cpp/src/arrow/csv/fuzz.cc Resolved
Comment thread cpp/src/arrow/csv/fuzz.cc Resolved
@pitrou

pitrou commented Oct 22, 2026

Copy link
Copy Markdown
Member

cc @addisoncrump if you want a bit more of fuzzing-related fun :)

Signed-off-by: Arthur Chan <arthur.chan@adalogics.com>
@arthurscchan

Copy link
Copy Markdown
Author

Thanks for the comments, I will fix them and reconsider some of the approach soon. I will also handles the changes on OSS-Fuzz side.

For the PR message, I am sorry this is my first time committing to this repository and I missed the template requirements, I will fix them, together with the other two PRs when I fix and address your comments.

@addisoncrump

addisoncrump commented Oct 23, 2026 •

Copy link
Copy Markdown

Honestly, this might be a great use case for libfuzzer's support for custom mutations. One could use the random seed to decide whether to mutate a subset structurally or the whole input. When the former: parse the CSV, pick some random subset of entries, un-escape them, mutate them, re-escape them, put them back. Otherwise, just use the normal mutator. Would also need to define a custom crossover, though.

(though, admittedly, this is a lot of extra work with maybe not a ton of gain 😅 CSV is already fairly unstructured)

Signed-off-by: Arthur Chan <arthur.chan@adalogics.com>
Copilot AI balanced review requested due to automatic review settings November 19, 2026 14:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are identified for the CSV fuzzing changes.

No open issues 🎉

What changed in this PR

This PR expands Arrow’s C++ CSV fuzzing coverage for GH-54903 by varying reader options, comparing reader results, and improving seed corpus and dictionary preparation.

Changes:

  • Adds a packed options header while preserving legacy corpus input handling.
  • Compares results across serial, threaded, and streaming readers and generates seeds for multiple CSV dialects.
  • Skips duplicate corpus content and packages CSV fuzzing dictionaries.
File Description
cpp/​src/​arrow/​csv/​generate_fuzz_corpus.cc Generates header-prefixed seeds for multiple CSV dialects.
cpp/​src/​arrow/​csv/​fuzz.cc Decodes reader options and adds differential table checks.
cpp/​src/​arrow/​csv/​fuzz_header_internal.h Defines the packed options header and serialization helpers.
cpp/​build-support/​fuzzing/​pack_corpus.py Warns and skips files with duplicate content.
cpp/​build-support/​fuzzing/​generate_corpuses.sh Copies dictionaries to the output and documents their use.
cpp/​build-support/​fuzzing/​dicts/​arrow-csv-fuzz.dict Adds CSV syntax and edge-case tokens as mutation hints.

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants