Add targetDocsPerChunk cap to VarByteChunkForwardIndexWriterV6 and wire it to table config

[xiangfu0]

Summary

VarByteChunkForwardIndexWriterV6 currently closes a chunk only when the next entry would
overflow the chunkSize-byte buffer. This PR lets a chunk additionally be bounded by document
count, and wires that control through to table config.

• New 4-arg constructor VarByteChunkForwardIndexWriterV6(file, compressionType, chunkSize, targetDocsPerChunk). The existing 3-arg constructor delegates with targetDocsPerChunk = -1
  (DISABLE_DOCS_PER_CHUNK).
• When targetDocsPerChunk > 0, a chunk is flushed once it holds that many docs, even if the byte
  buffer isn't full; otherwise behavior is unchanged.
• The buffer-overflow flush predicate is extracted into a protected shouldStartNewChunk(int) hook
  in VarByteChunkForwardIndexWriterV4 (mirroring the existing writeChunkHeader hook), so V6 adds
  the cap without duplicating putBytes().
• New table-config option enforceTargetDocsPerChunk (default false) promotes the existing
  targetDocsPerChunk from a byte-size input into a hard per-chunk document cap.

Motivation

The table-config targetDocsPerChunk does not actually control documents per chunk. It only
derives the chunk byte budget:

chunkSize = max(min(maxLength * targetDocsPerChunk, targetMaxChunkSize), 4KB)

Because that keys off the column's longest value, a column whose max length far exceeds its
average length clamps to targetMaxChunkSize and stops tracking the requested doc count entirely.

Measured on a 1M-doc column of short log lines (avg 106 B) with 0.1% ~50 KB stack traces
(max 50094 B), targetMaxChunkSize=1MB:

targetDocsPerChunk	derived chunkBytes	actual docs/chunk
50	1 MB (clamped)	9345
250	1 MB (clamped)	9345
1000 (the default)	1 MB (clamped)	9345
5000	1 MB (clamped)	9345

Every setting yields the same 9,345 docs per chunk — the default overshoots its own nominal target
by more than 9x. With enforceTargetDocsPerChunk: true each setting yields exactly N.

Since a point lookup decompresses a whole chunk, docs-per-chunk is the dominant term in random
access latency, so losing control of it is expensive.

Benchmark

BenchmarkVarByteV6TargetDocsPerChunk (added in this PR) sweeps the cap over index size,
sequential scan, 1%-selectivity ascending scan, and uniform random point lookups. 1M docs, ZSTD,
chunkSize=1MB, JDK 25, Apple Silicon.

LOW_CARD_URL (avg 73 B, 200 distinct values):

N	docs/chunk	size	vs uncapped	random lookup	1% asc scan	full scan
-1	13513	2.88 MB	1.00x	199.0 µs	1.52 µs/doc	26.1 ms
50	50	5.93 MB	2.06x	3.45 µs	3.33	77.3
250	250	3.76 MB	1.31x	6.85 µs	2.62	37.2
1000	1000	3.24 MB	1.12x	17.8 µs	1.78	29.4
5000	5000	2.97 MB	1.03x	79.5 µs	1.57	26.0

JSON_LOG (avg 163 B, structurally repetitive, unique payloads):

N	docs/chunk	size	vs uncapped	random lookup	1% asc scan	full scan
-1	6250	34.26 MB	1.00x	495.3 µs	7.98 µs/doc	100.9 ms
50	50	38.00 MB	1.11x	7.18 µs	7.05	162.3
250	250	33.61 MB	0.98x	22.1 µs	8.76	109.0
1000	1000	36.28 MB	1.06x	101.0 µs	10.08	121.0
5000	5000	34.20 MB	1.00x	392.2 µs	7.98	99.4

SKEWED_LOG (avg 106 B, max 50094 B):

N	docs/chunk	size	vs uncapped	random lookup	1% asc scan	full scan
-1	9345	8.15 MB	1.00x	335.9 µs	3.56 µs/doc	51.6 ms
50	50	11.63 MB	1.43x	4.62 µs	4.51	106.0
250	250	9.24 MB	1.13x	12.1 µs	4.84	63.9
1000	1000	8.73 MB	1.07x	38.3 µs	3.96	55.6
5000	5000	8.30 MB	1.02x	174.8 µs	3.61	52.3

Takeaways:

• Random point lookups scale ~linearly with docs/chunk. Uncapped to N=250 is a 22–29x latency
  reduction across all three datasets.
• The size cost at N=250 is small to nil: 1.31x, 0.98x, 1.13x. Compression ratio is not
  monotonic in chunk size — JSON_LOG bottoms out at N=250–500, and HIGH_CARD_UUID at N=1000
  (18.83 MB vs 20.49 MB at N=5000). So there is a real tunable sweet spot rather than
  "larger is always smaller".
• Small caps overshoot. N=50 buys only 2–3x over N=250 on lookups but costs 1.11–2.06x size and
  1.5–3x on full scan.
• This is a point-lookup optimization, not a general one. The 1% ascending scan (the typical
  filtered-query shape) is flat-to-slightly-worse at every cap, since decompression already
  amortizes over ~135 values per uncapped chunk. Full scan regresses 8–43% at N=250.

N≈250 looks like a reasonable starting point for point-lookup-heavy columns.

Caveats: random-access error bars are wide at large chunk sizes (JSON_LOG uncapped: 495 ± 399 µs,
Cnt=3) — direction and order of magnitude are solid, exact values approximate. All measurements
are warm-buffer; on a server with mmap'd segments and cold pages, smaller chunks should also cut
read amplification, so the lookup win is likely understated. Latency swept on ZSTD only.

Configuration

{
  "fieldConfigList": [
    {
      "name": "myColumn",
      "encodingType": "RAW",
      "indexes": {
        "forward": {
          "compressionCodec": "ZSTANDARD",
          "rawIndexWriterVersion": 6,
          "targetMaxChunkSize": "1MB",
          "targetDocsPerChunk": 250,
          "enforceTargetDocsPerChunk": true
        }
      }
    }
  ]
}

• rawIndexWriterVersion: 6 selects this writer. enforceTargetDocsPerChunk has no effect on
  earlier versions, which have no cap mechanism.
• Without enforceTargetDocsPerChunk, targetDocsPerChunk keeps its existing byte-size-derivation
  meaning exactly.
• ForwardIndexConfig.setDefaultEnforceTargetDocsPerChunk allows setting it cluster-wide, matching
  the existing targetDocsPerChunk pattern.
• Enforcement with a non-positive targetDocsPerChunk is rejected at config construction.

Note the chunk buffer is still allocated at chunkSize regardless of the cap, so a small cap
against a 1 MB targetMaxChunkSize leaves most of the buffer unused. Lower targetMaxChunkSize
alongside the cap; nothing auto-derives it.

Constructor examples

// Default (3-arg): byte-driven flush only — identical to the prior behavior.
new VarByteChunkForwardIndexWriterV6(file, ChunkCompressionType.ZSTANDARD, chunkSize);

// Cap each chunk at 1000 docs: flush at 1000 docs OR when the chunkSize buffer
// fills, whichever comes first.
new VarByteChunkForwardIndexWriterV6(file, ChunkCompressionType.ZSTANDARD, chunkSize, 1000);

// Explicitly disable the docs cap (equivalent to the 3-arg constructor).
new VarByteChunkForwardIndexWriterV6(file, ChunkCompressionType.ZSTANDARD, chunkSize,
    VarByteChunkForwardIndexWriterV6.DISABLE_DOCS_PER_CHUNK);

Backward compatibility

• The defaults (-1 at the writer, enforceTargetDocsPerChunk: false at the config) reproduce the
  exact current behavior. All pre-existing constructor and factory signatures are preserved as
  delegating overloads.
• The on-disk format and writer version (6) are unchanged.
• The cap is a write-time decision only. The chunk size lives in the index file header and the
  chunk boundaries in the per-chunk metadata, so readers derive all geometry from the segment and
  never consult table config. Retuning affects only newly built segments; segments with different
  geometries remain readable side by side, so the setting can be tuned over time without a reload.

Testing

• EnforceTargetDocsPerChunkTest drives a skewed column (avg ~18 B, max 50 KB) through
  SingleValueVarByteRawIndexCreator and asserts enforcement yields exactly numDocs / target
  chunks while the unenforced path yields far fewer — pinning the gap the flag closes — plus a value
  round-trip on both paths.
• VarByteChunkV6Test#testTargetDocsPerChunkCapsChunk asserts each capped chunk holds exactly
  targetDocsPerChunk docs and that values round-trip.
• New ForwardIndexConfigTest cases cover the default, JSON round-trip, builder copy, and rejection
  of enforcement with a non-positive target; new ForwardIndexUtilsTest cases cover
  getDocsPerChunkCap.
• 2,735 tests pass across the forward-index creator/loader/writer suites in pinot-segment-local,
  plus 20 in pinot-segment-spi.

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

❌ Patch coverage is 84.61538% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.51%. Comparing base (81b82b2) to head (8a94f20).

Files with missing lines	Patch %	Lines
...ment/index/forward/ForwardIndexCreatorFactory.java	33.33%	4 Missing ⚠️
...he/pinot/segment/spi/index/ForwardIndexConfig.java	78.57%	3 Missing ⚠️
.../writer/impl/VarByteChunkForwardIndexWriterV6.java	83.33%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #18772      +/-   ##
============================================
+ Coverage     65.50%   65.51%   +0.01%     
  Complexity     1421     1421              
============================================
  Files          3430     3430              
  Lines        218000   218042      +42     
  Branches      34642    34646       +4     
============================================
+ Hits         142799   142854      +55     
+ Misses        63641    63627      -14     
- Partials      11560    11561       +1     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (ø)
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-25	65.51% <84.61%> (+0.01%)	⬆️
temurin	65.51% <84.61%> (+0.01%)	⬆️
unittests	65.51% <84.61%> (+0.01%)	⬆️
unittests1	56.80% <40.38%> (+<0.01%)	⬆️
unittests2	37.88% <63.46%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Adds an optional document-count cap for chunk flushing in VarByteChunkForwardIndexWriterV6, allowing callers to control chunk granularity by both byte budget (chunkSize) and a new targetDocsPerChunk parameter, while keeping the on-disk format/version unchanged.

Changes:

• Introduces a protected shouldStartNewChunk(int sizeRequired) hook (and a helper for current-chunk doc count) in V4 so subclasses can add extra chunk-flush predicates without duplicating putBytes().
• Extends V6 with a 4-arg constructor and a docs-per-chunk cap that triggers chunk flushes once the current chunk reaches the configured doc count.
• Adds a V6 unit test intended to validate docs-per-chunk behavior and round-trip correctness.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/io/writer/impl/VarByteChunkForwardIndexWriterV4.java	Extracts the “start new chunk” predicate into an overridable hook and exposes current-chunk doc count for subclasses.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/io/writer/impl/VarByteChunkForwardIndexWriterV6.java	Adds optional targetDocsPerChunk support via shouldStartNewChunk override and a new constructor overload.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/creator/VarByteChunkV6Test.java	Adds a test for docs-per-chunk capping and value round-tripping.