Repository navigation
GH-54903: [C++][OSS-Fuzz fuzzing] Update csv fuzzer and corpora/dict preparation - #54912
arthurscchan wants to merge 3 commits into
Conversation
Signed-off-by: Arthur Chan <arthur.chan@adalogics.com>
|
Thanks for the PR @arthurscchan . Is there any reason this doesn't reuse our standard PR description template? |
pitrou
left a comment
There was a problem hiding this comment.
Thanks for doing this. On the principle this is interesting, but I think there are number of problems with the current implementation.
|
cc @addisoncrump if you want a bit more of fuzzing-related fun :) |
Signed-off-by: Arthur Chan <arthur.chan@adalogics.com>
|
Thanks for the comments, I will fix them and reconsider some of the approach soon. I will also handles the changes on OSS-Fuzz side. For the PR message, I am sorry this is my first time committing to this repository and I missed the template requirements, I will fix them, together with the other two PRs when I fix and address your comments. |
|
Honestly, this might be a great use case for libfuzzer's support for custom mutations. One could use the random seed to decide whether to mutate a subset structurally or the whole input. When the former: parse the CSV, pick some random subset of entries, un-escape them, mutate them, re-escape them, put them back. Otherwise, just use the normal mutator. Would also need to define a custom crossover, though. (though, admittedly, this is a lot of extra work with maybe not a ton of gain 😅 CSV is already fairly unstructured) |
Signed-off-by: Arthur Chan <arthur.chan@adalogics.com>
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are identified for the CSV fuzzing changes.
No open issues 🎉
What changed in this PR
This PR expands Arrow’s C++ CSV fuzzing coverage for GH-54903 by varying reader options, comparing reader results, and improving seed corpus and dictionary preparation.
Changes:
- Adds a packed options header while preserving legacy corpus input handling.
- Compares results across serial, threaded, and streaming readers and generates seeds for multiple CSV dialects.
- Skips duplicate corpus content and packages CSV fuzzing dictionaries.
| File | Description |
|---|---|
| cpp/src/arrow/csv/generate_fuzz_corpus.cc | Generates header-prefixed seeds for multiple CSV dialects. |
| cpp/src/arrow/csv/fuzz.cc | Decodes reader options and adds differential table checks. |
| cpp/src/arrow/csv/fuzz_header_internal.h | Defines the packed options header and serialization helpers. |
| cpp/build-support/fuzzing/pack_corpus.py | Warns and skips files with duplicate content. |
| cpp/build-support/fuzzing/generate_corpuses.sh | Copies dictionaries to the output and documents their use. |
| cpp/build-support/fuzzing/dicts/arrow-csv-fuzz.dict | Adds CSV syntax and edge-case tokens as mutation hints. |
Review effort: Balanced
Rationale for this change
This PR improves the fuzzing coverage and effectiveness of the Arrow C++ CSV reader, as part of the broader effort described in GH-54903. Currently, the CSV fuzz target reads every input with a fixed set of hardcoded options, and its seed corpus only covers a single dialect, so most of the reader's option space (delimiters, quoting/escaping, null handling, block sizes, dictionary encoding, timestamp parsers, ...) is explored mostly by chance.
This change is coordinated with the OSS-Fuzz side in google/oss-fuzz#16332.
What changes are included in this PR?
cpp/src/arrow/csv/fuzz.cc:TableReader, multi-threadedTableReader,StreamingReader, andCountRowsAsync) is given a fresh input stream, since a reader consumes the stream it is passed.cpp/src/arrow/csv/fuzz_header_internal.h(new): the packedFuzzCsvHeaderstructure (bit-field option groups withstatic_asserts on the layout), and its serialization and parsing/validation helpers.cpp/src/arrow/csv/generate_fuzz_corpus.cc: generated seed files are now prefixed with the serialized fuzz header matching the dialect used for writing, and several dialect variants are generated (alternate delimiters, quote-all, headerless with autogenerated column names,N/Anulls). The batch-size variant was dropped, as it only exercised the writer's internal chunking, which is irrelevant for the reader.cpp/build-support/fuzzing/pack_corpus.py: files with duplicate content now emit a warning and are skipped instead of failing the corpus packing (some Pandas test CSVs are exact duplicates of each other).cpp/build-support/fuzzing/generate_corpuses.sh: removed the now-unneeded deduplication workaround; documented how dictionaries are used (they are hints to the fuzzing engine, not restrictions) - also documented incpp/build-support/fuzzing/dicts/arrow-csv-fuzz.dict.Are these changes tested?
Yes.
Are there any user-facing changes?
No.
Was AI used for this PR?
In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.
PR code and description written by:
Reviewed before submission by:
Code and description were drafted with the help of an AI coding agent, then reviewed, revised and validated by the author before submission.