Skip to content

GH-55487: [C++][Parquet] Batch length encoding and bulk-copy payloads in DeltaLengthByteArrayEncoder::PutBinaryArray - #55488

Open
ryanvickr wants to merge 1 commit into
apache:mainfrom
ryanvickr:gh-55487-fix-dlba-encoder
Open

ryanvickr wants to merge 1 commit into
apache:mainfrom
ryanvickr:gh-55487-fix-dlba-encoder

Conversation

@ryanvickr

@ryanvickr ryanvickr commented Nov 10, 2026 •

Copy link
Copy Markdown

Rationale for this change

DeltaLengthByteArrayEncoder::PutBinaryArray currently encodes Arrow binary/string lengths one value at a time and appends payloads without pre-reserving sink_, whereas Put(const ByteArray*, int) batches lengths and pre-reserves capacity.

The memory pool and zero-length UnsafeAppend fixes previously in this PR have been split out into #55686.

What changes are included in this PR?

  • Batch length encoding in 256-value chunks and pre-reserve sink_ capacity in DeltaLengthByteArrayEncoder::PutBinaryArray.
  • Add a fast path for non-null offset-based binary/string arrays that bulk-copies the contiguous payload range in a single append while preserving offset/overflow validation.

Are these changes tested?

Yes, added unit tests in cpp/src/parquet/encoding_test.cc covering batch boundaries, slices, StringViewArray, multi-Put accumulation, and invalid/overflowing offset rejection.

Are there any user-facing changes?

No.

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

Closes #55487

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #55487 has been automatically assigned in GitHub to PR creator.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #55487 has no components, please add labels for components.

@pitrou

pitrou commented Nov 16, 2026

Copy link
Copy Markdown
Member

@ryanvickr Can you please rewrite the PR description to be more terse and higher-level than the current AI-generated details? Thanks in advance.

@pitrou

pitrou commented Nov 16, 2026

Copy link
Copy Markdown
Member

@ryanvickr I think it would be worth doing a separate PR for the two simple fixes (pass memory pool and avoid calling UnsafeAppend on zero-length strings). Can you do that so that the performance optimization can be discussed independently?

…yloads in DeltaLengthByteArrayEncoder::PutBinaryArray

Closes apache#55487
@ryanvickr
ryanvickr force-pushed the gh-55487-fix-dlba-encoder branch from 9d41a76 to cc04079 Compare November 20, 2026 03:59
@ryanvickr ryanvickr changed the title GH-55487: [C++][Parquet] Fix DeltaLengthByteArrayEncoder memory pool tracking and Arrow string batching GH-55487: [C++][Parquet] Batch length encoding and bulk-copy payloads in DeltaLengthByteArrayEncoder::PutBinaryArray Nov 20, 2026
@ryanvickr

Copy link
Copy Markdown
Author

Thanks for the review @pitrou! I split the memory pool and zero-length UnsafeAppend fixes out into #55686, narrowed this PR to the PutBinaryArray optimization, and tightened the PR descriptions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[C++][Parquet] Optimize DeltaLengthByteArrayEncoder::PutBinaryArray with batched length encoding and bulk copy

2 participants