Skip to content

GH-55669: [C++][IPC] Validate buffer bounds and replace raw pointer arithmetic in MessageDecoder - #55687

Open
Mr-Nobody-Anonymous wants to merge 8 commits into
apache:mainfrom
Mr-Nobody-Anonymous:fix-pr-55671-ipc-bounds
Open

Mr-Nobody-Anonymous wants to merge 8 commits into
apache:mainfrom
Mr-Nobody-Anonymous:fix-pr-55671-ipc-bounds

Conversation

@Mr-Nobody-Anonymous

@Mr-Nobody-Anonymous Mr-Nobody-Anonymous commented Nov 20, 2026 •

Copy link
Copy Markdown

Rationale for this change

In cpp/src/arrow/ipc, streaming decoding and decompression can be hardened against malformed or untrusted buffers (GH-55669). This pull request incorporates the hardening work originally proposed by @Awesome06 in #55671, while resolving the build failures across the C++ matrix and addressing all active review findings.

What changes are included in this PR?

  1. Memory Safety and Bound Checks in MessageDecoder:

    • Replaces raw pointer arithmetic and manual memcpy with std::span and checked offsets in MessageDecoderImpl.
    • Validates input sizes against negative values in MessageDecoder::Consume.
    • Protects chunk reassembly against out-of-bounds copies.
  2. IPC Read Ceilings in IpcReadOptions:

    • Adds configurable max_metadata_length, max_body_length, and max_decompression_ratio to IpcReadOptions.
    • Rejects negative lengths in CheckMetadataLength and CheckBodyLength.
    • Enforces configured limits in the shared ReadRecordBatchInternal function so that direct public ReadRecordBatch entry points cannot bypass bounds checks.
  3. FlatBuffers and Metadata Hardening:

    • Adds null checks for FlatBuffers tensor and custom metadata fields.
    • Removes unconditional metadata->reserve in GetKeyValueMetadata to prevent untrusted metadata counts from triggering premature large vector allocations.
  4. CI Build and Test Fixes:

    • Fixes compiler deduction error on read_write_test.cc:3579 by separating int64_t* and int32_t* options in initializer lists.
    • Uses public IPC APIs (MessageReader::Open, ReadSchema) in direct record batch regression tests instead of internal RecordBatchFileReader helpers.

Are these changes tested?

Yes, comprehensive unit tests have been added to cpp/src/arrow/ipc/read_write_test.cc covering:

  • Malformed header tokens and negative continuation tokens.
  • Negative read limit options validation.
  • Metadata and body length limit enforcement in streaming and direct record-batch reads.
  • Decompression ratio limits.

Are there any user-facing changes?

New optional fields in IpcReadOptions: max_metadata_length, max_body_length, and max_decompression_ratio. Defaults remain -1 (unlimited), preserving backwards compatibility.

Was AI used for this PR?

PR code and description written by:

  • AI

Reviewed before submission by:

  • Human

Awesome06 and others added 8 commits November 20, 2026 07:28
…in MessageDecoder

* Replace raw pointer arithmetic and unchecked memcpy offsets in MessageDecoderImpl with std::span and AddWithOverflow

* Enforce validation against negative data size in MessageDecoder::Consume

* Add opt-in max_metadata_length, max_body_length, and max_decompression_ratio ceilings to IpcReadOptions (defaulting to unlimited: -1)

* Use CHECK_FLATBUFFERS_NOT_NULL guards in Tensor metadata decoders

* Add deterministic unit tests covering negative sizes, opt-in limits, and decompression bounds in read_write_test
…r and enforce reader limits

* Avoid use-after-free in MessageDecoderImpl when listener OnEOS resets the decoder
* Early exit when data span is fully consumed before trailing member dereferences
* Wire IpcReadOptions through ReadMessage, MessageReader, and RecordBatchFileReader
* Add block bounds validation and test coverage for IPC reader limits
…force file footer bound

* Add IpcReadOptions::Validate() to reject negative values below -1
* Wire options validation to stream and file reader entry points
* Enforce max_metadata_length against footer_length in ReadFooterAsync
* Add tests for footer bounds, negative options validation, and decoder rejection
… MessageReader streams

* Validate metadata and body lengths in StreamDecoderInternal::OnMessageDecoded
* Protect custom MessageReader pathways against unbounded allocations
* Add unit test coverage with MockCustomMessageReader
…ds tests

Consolidate repeated IPC metadata and body length validations into IpcReadOptions, fix coalescing in generator limits test, and deduplicate bounds test cases.

- Add CheckMetadataLength and CheckBodyLength helper methods to IpcReadOptions to centralize limit validation and error message formatting.
- Replace repetitive validation blocks across message.cc and reader.cc with IpcReadOptions check methods.
- Update TestFileReaderLimits.GeneratorCoalesceBlockLimitsExceeded to use NoZeroCopyBufferReader to trigger coalesced range validation.
- Consolidate bounds and limits tests in read_write_test.cc to reduce boilerplate and eliminate redundant test coverage.
…den bounds checks

- Fix template deduction failure in NegativeLimitsInvalid test by separating int64_t* and int32_t* fields
- Enforce negative length rejection in CheckMetadataLength and CheckBodyLength
- Enforce metadata and body read limits in shared ReadRecordBatchInternal
- Remove unconditional metadata reserve in GetKeyValueMetadata
- Add unit tests for negative lengths and direct ReadRecordBatch limits
Copilot AI balanced review requested due to automatic review settings November 20, 2026 04:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@Awesome06

Copy link
Copy Markdown
Contributor

Hi @Mr-Nobody-Anonymous,

Thank you for your interest in this IPC hardening work. However, please note that #55671 is the original, actively maintained pull request for GH-55669.

The compiler template deduction issue on read_write_test.cc:3579 has already been resolved directly on #55671, and CI is running on the updated branch.

Opening a duplicate pull request that re-packages an active PR creates unnecessary overhead for the Apache Arrow maintainers and fragments the review process. Under Apache community guidelines, active work should remain consolidated on the original PR unless the author has abandoned it (which is not the case here).

Please close this duplicate pull request (#55687) so review efforts remain focused on #55671. If you have review feedback or ideas for separate follow-up hardening (such as the metadata->reserve change, which falls outside the original MessageDecoder scope), you are welcome to share them as review comments on #55671 or open a separate follow-up issue once the primary fix lands.

Thank you!

@rok

rok commented Nov 20, 2026

Copy link
Copy Markdown
Member

Hi @Awesome06 and @Mr-Nobody-Anonymous. Thank you for opening these PRs and coordinating here to guide them to some outcome. As an Arrow contributor I am really curious about your motivation for working on IPC hardening.
Could you say what motivates you (your own need, your employers, just curiosity,..) to work on this and what's your plan going forward?
The reason I ask is that as a project Arrow has had an influx of new contributors and I'm curious why.

@Awesome06

Copy link
Copy Markdown
Contributor

Hey @rok, Thanks for asking! I'm a CS undergrad, so for me it's mostly curiosity about how data tools work under the hood and wanting hands-on experience working on real, large codebases. That's why I've been contributing across Arrow's compute and Parquet code as well as DataFusion.

As for the influx of new contributors, I think a few things are colliding at once. Arrow has become the backbone for so many modern databases and query engines that a lot of students (including folks starting early ahead of GSoC) want experience here, and newer dev tools have made it much easier to navigate a big C++ repo than it used to be. Specifically on the IPC side, Google's Patch Rewards program covers OSS-Fuzz projects like Arrow, so getting a payout as a side bonus for catching parser bugs definitely explains the recent spike in PRs there.

Going forward, I don't want to limit myself to just one area. I plan to contribute to a few different open-source projects to become a more well-rounded engineer, while keeping any PRs I open here small, well-tested, and easy to review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants