Skip to content

GH-34319: [R] Fast path for plain R metadata sanitization - #55673

Draft
Aurantium-47 wants to merge 1 commit into
apache:mainfrom
Aurantium-47:perf/r-metadata-sanitizer-fastpath
Draft

Aurantium-47 wants to merge 1 commit into
apache:mainfrom
Aurantium-47:perf/r-metadata-sanitizer-fastpath

Conversation

@Aurantium-47

@Aurantium-47 Aurantium-47 commented Nov 18, 2026 •

Copy link
Copy Markdown

Rationale for this change

Related to #34319; separate from attribute-restoration PR #55668.

@thisisnic's analysis in #34319 already identified deserializing and sanitizing the full R metadata as the main cost when selecting only a few columns from a wide file. That comment explicitly discloses AI-assisted analysis. This draft investigates that first work item; it does not claim to discover the bottleneck or bypass metadata checks for unselected columns.

The trade-off is material: remeasurement of the final head improves the synthetic narrow-projection benchmarks by 3.12× (20 columns) and 2.93× (200 columns), while metadata with a complex attribute in the final column makes the sanitizer-only fallback 20.7% slower. This draft is for review of that trade-off and the safety boundary, not a claim that every metadata workload improves.

What changes are included in this PR?

  • Check the entire metadata tree for a deliberately narrow safe shape: plain lists, optional plain character names, and unattributed allowed atomic/NULL leaves.
  • Return that tree unchanged only after all of it passes. Check list-element types before extraction, including promises.
  • Retain the original filtering and warning/error path for other types, custom attributes/list subclasses, or preflight depth above 64. No deserialization, ASCII, gzip or promise-entry gate is removed.
  • Preserve on_save argument evaluation and avoid dispatching list class methods during preflight.
  • Add regression coverage for names/empty elements, unsafe warnings and their ordering, malformed matrix error context, custom attributes/deep trees, serialized promises, class method dispatch and delayed on_save evaluation.

The original protection in #41969 and list-subclass handling in #43895 remain in place. There is no API or file-format change.

Performance, including the regression

Final-head remeasurement

The R code at final head b5773cd280674c0ce0816c8e2e601c0d3ba554a3 was remeasured with R 4.5.3 and the Arrow native library held fixed at 25.0.1. Each workload below used eight alternating original/candidate trials, with matching outputs. The file-read workloads use a synthetic 16,000-column Parquet file; attribute restoration is held fixed, so these results do not combine this patch with #55668.

Workload Original median Final-head candidate median Result
Read 20 of 16,000 columns 1.8355 s 0.5875 s 3.12× faster
Read 200 of 16,000 columns 1.7800 s 0.6070 s 2.93× faster
Sanitize with a complex attribute in the final column 1.3470 s 1.6260 s 20.7% slower

The last row measures the sanitizer alone: a late fallback pays for both the preflight and the original walk. The positive and negative results are both part of the trade-off. This rerun is not a full current-main native/source package build or a real production workload, and it does not resolve the other unvalidated items listed below.

Historical measurements (retained for traceability)

Linux x86_64, R 4.5.3, conda-forge Arrow 25.0.1 native library. Synthetic 16,000-column × 10-row Parquet file, with text and levels attributes on each column. Original attribute restoration is held fixed; this is not a measurement of #55668 combined with this patch. Six alternating trials per end-to-end case, garbage collection before each trial, output equality checked.

Workload Original median Candidate median Result
Read 20 of 16,000 columns 1.9650 s 0.6745 s 2.91× faster
Read 200 of 16,000 columns 1.8195 s 0.6225 s 2.92× faster
Sanitize plain 16,000-column metadata 1.465 s 0.277 s 5.29× faster
Sanitize with a complex attribute in the final column 1.406 s 1.718 s 22.2% slower

The last two rows use three trials. A late fallback pays for both the preflight and the original walk. Batched, warmed small-input tests likewise show fallback regressions of about 9%, 15% and 21% at 1, 20 and 200 columns respectively. These are synthetic examples, not a general Arrow speedup claim.

ASCII metadata decode plus sanitization measured 1.835 → 0.514 s; gzip-wrapped decode plus sanitization measured 1.727 → 0.452 s (three trials each, identical results). These are metadata-only measurements, not full compressed-Parquet reads.

Recorded allocations for the wide sanitizer decreased from 640,288 to 384,192 bytes on the plain path but increased to 1,024,480 bytes on the fallback path. Whole-process peak RSS did not improve: approximately 167 → 168 MiB on the plain path and 167 → 170 MiB on fallback, one isolated process per condition. No memory improvement is claimed.

The timing series preceded the final fixes for rejecting custom classes without calling names(x) and preserving on_save evaluation; the extended sanitizer/compression series includes the former fix. Correctness and the peak-RSS measurements were rerun on the final code. These historical timings should be reconfirmed by CI/review rather than treated as final-code release benchmarks.

Are these changes tested?

Passed locally on the final candidate:

  • 2,092 original/candidate comparisons of returned values, warning text/order and error class/message, including random nested fixtures, unsafe options, custom attributes, ALTREP sequences and depths through 500 under default limits.
  • Nested and top-level serialized promise checks in safe/unsafe modes; the harmless stop() fixture was not executed.
  • 105 existing expectations from the working Arrow checkout's metadata tests plus 19 new assertions, with zero failures/errors/skips. The candidate was applied in a separate exported checkout and its R source loaded against the installed Arrow 25.0.1 runtime.
  • Patch application and whitespace checks against the freshly fetched main r/R/metadata.R.

Review caught and corrected three initial problems: replacing recursive purrr::map() changed indexed error context; the first guard could invoke names() methods on a classed list; and the initial early return skipped on_save evaluation. The first design was discarded; the latter two have explicit regressions.

Not yet validated:

  • A full current-main native/source package build, full package tests or R CMD check.
  • Older supported R versions. Only R 4.5.3 was exercised; R 4.4 environment provisioning did not complete. R 4.5.3 itself rejects a top-level promise, so the older-R entry guard is not independently verified here.
  • Equivalent behavior under deliberately reduced expression/stack limits. A controlled stack-exhaustion experiment was interrupted while the original error path was taking too long; it is not counted as passing.
  • Real-world frequency/acceptability of the late-fallback regression.

Was AI used for this PR?

AI was used for investigation, implementation, differential review, regression tests, benchmark execution and this description. No human review is claimed.

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

Check the whole metadata tree before returning plain safe metadata unchanged;
retain the existing filtering and error path for all other shapes.

AI-assisted implementation and review. The supplied benchmark documents both
the narrow-projection gain and the late-fallback regression; no human review
is claimed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant