feat(reader): read vortex.parquet.variant (#445)

CHANGELOG.md

@@ -13,7 +13,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `fastlanes.delta` and `vortex.patched` belong to no edition, as in Rust: emit them with the new `WriteOptions.withoutEditions()`, the counterpart of Rust's `disable_editions()` ([#441](https://github.com/dfa1/vortex-java/issues/441)).
 - Nullable low-cardinality string columns dict-encoded per chunk store null as a dictionary entry, as Rust does, instead of adding a row validity bitmap: about 20% smaller, now slightly below vortex-jni ([2091beb](https://github.com/dfa1/vortex-java/commit/2091beb3)).
 
+### Added
+- Read `vortex.parquet.variant`, the encoding vortex-jni's default writer uses for Arrow `arrow.parquet.variant` columns: each row's Apache Variant `metadata`/`value` binaries come back as a struct ([#445](https://github.com/dfa1/vortex-java/issues/445)).
+
 ### Fixed
+- Dictionary-encoded Binary columns (e.g. the `value` child of `vortex.parquet.variant`) failed to read with a `ClassCastException` ([#445](https://github.com/dfa1/vortex-java/issues/445)).
 - Null rows of Rust-written `vortex.list` columns read as empty lists: the decoder ignored the list's validity child.
 - CSV export (`CsvExporter`, `vortex export`) failed on list-view columns with `unsupported array type for CSV export: ListViewArray`.
 - `vortex inspect --html` grouped the Chunks panel by chunk index, so files whose columns chunk differently (e.g. Rust's `tpch_orders.compact`) showed overlapping row ranges and mixed sizes; the panel now lists one entry per distinct row range.

core/src/main/java/io/github/dfa1/vortex/core/model/Editions.java

@@ -23,10 +23,10 @@
 /// zone-map aggregates) has no members here. Encodings in no edition at all — `fastlanes.delta`,
 /// `vortex.patched` — are writable only with the guard turned off, as in Rust.
 ///
-/// vortex-java implements every `core`-family encoding except `vortex.parquet.variant`, and not
-/// `zstd`'s `vortex.zstd_buffers`; both have no [EncodingId.WellKnown] constant yet and are named
-/// as [EncodingId.Custom] instead: the
-/// catalog mirrors upstream faithfully rather than being truncated to what is implemented today.
+/// vortex-java implements every `core`-family encoding (`vortex.parquet.variant` read only), but not
+/// `zstd`'s `vortex.zstd_buffers`, which has no [EncodingId.WellKnown] constant yet and is named as
+/// an [EncodingId.Custom] instead: the catalog mirrors upstream faithfully rather than being
+/// truncated to what is implemented today.
 public final class Editions {
 
     /// The baseline `core` edition: stable encodings writable by Vortex (Rust reference) 0.36.0.
@@ -75,7 +75,7 @@ public final class Editions {
     /// `core` edition, and the one the default writer targets, as in Rust.
     public static final Edition CORE_2026_08_3 = new Edition(
             new EditionId(EditionFamily.CORE, YearMonth.of(2026, 8), 3),
-            Set.of(new EncodingId.Custom("vortex.parquet.variant"), EncodingId.VORTEX_VARIANT));
+            Set.of(EncodingId.VORTEX_PARQUET_VARIANT, EncodingId.VORTEX_VARIANT));
 
     /// The August 2026 draft edition of the `preview` family. Empty in Rust too: no component has
     /// entered preview yet.

core/src/main/java/io/github/dfa1/vortex/core/model/EncodingId.java

@@ -112,6 +112,9 @@ enum WellKnown implements EncodingId {
         VORTEX_PATCHED("vortex.patched"),
         /// Variant logical encoding: canonical container over `core_storage` plus an optional shredded child.
         VORTEX_VARIANT("vortex.variant"),
+        /// Parquet Variant physical encoding (`vortex.parquet.variant`): per-row Apache Variant binary
+        /// `metadata` and `value` children, plus an optional shredded `typed_value` child.
+        VORTEX_PARQUET_VARIANT("vortex.parquet.variant"),
         ;
 
         // O(1) access to a WellKnown constant by its string representation
@@ -253,4 +256,6 @@ public String toString() {
     WellKnown VORTEX_PATCHED = WellKnown.VORTEX_PATCHED;
     /// Well-known `vortex.variant` id.
     WellKnown VORTEX_VARIANT = WellKnown.VORTEX_VARIANT;
+    /// Well-known `vortex.parquet.variant` id.
+    WellKnown VORTEX_PARQUET_VARIANT = WellKnown.VORTEX_PARQUET_VARIANT;
 }

core/src/main/java/io/github/dfa1/vortex/core/proto/ProtoParquetVariantMetadata.java

@@ -0,0 +1,73 @@
+package io.github.dfa1.vortex.core.proto;
+
+import java.io.IOException;
+import java.lang.foreign.MemorySegment;
+import javax.annotation.processing.Generated;
+
+/// Generated from proto3 message {@code vortex.encodings.ParquetVariantMetadata}.
+/// Do not edit by hand — regenerate via {@code ./mvnw generate-sources -pl core -P regenerate-sources}.
+/// @param has_value field tag 1
+/// @param typed_value_dtype field tag 2
+/// @param value_nullable field tag 3
+@Generated("io.github.dfa1.vortex.protogen.CodeGen")
+public record ProtoParquetVariantMetadata(
+        boolean has_value,
+        ProtoDType typed_value_dtype,
+        boolean value_nullable
+) {
+
+    /// Decodes a {@code vortex.encodings.ParquetVariantMetadata} from a slice of a memory segment.
+    /// @param __seg backing segment
+    /// @param __off start offset in bytes
+    /// @param __len payload length in bytes
+    /// @return decoded record
+    /// @throws IOException if the slice is malformed or truncated
+    public static ProtoParquetVariantMetadata decode(MemorySegment __seg, long __off, long __len) throws IOException {
+        ProtoReader r = new ProtoReader(__seg, __off, __len);
+        boolean has_value = false;
+        ProtoDType typed_value_dtype = null;
+        boolean value_nullable = false;
+        while (r.hasMore()) {
+            int tag = r.readVarint32();
+            switch (tag >>> 3) {
+                case 1 -> {
+                    has_value = r.readBool();
+                }
+                case 2 -> {
+                    MemorySegment __slice = r.readLenDelimSegment();
+                    typed_value_dtype = ProtoDType.decode(__slice, 0, __slice.byteSize());
+                }
+                case 3 -> {
+                    value_nullable = r.readBool();
+                }
+                default -> r.skipField(tag & 7);
+            }
+        }
+        return new ProtoParquetVariantMetadata(has_value, typed_value_dtype, value_nullable);
+    }
+
+    /// Encodes this record to a proto3-wire-format byte array.
+    /// @return encoded bytes
+    public byte[] encode() {
+        ProtoWriter w = new ProtoWriter();
+        encodeTo(w);
+        return w.toByteArray();
+    }
+
+    void encodeTo(ProtoWriter w) {
+        if (has_value) {
+            w.writeTag(1, 0);
+            w.writeBool(has_value);
+        }
+        if (typed_value_dtype != null) {
+            w.writeTag(2, 2);
+            int __mark = w.beginLenDelim();
+            typed_value_dtype.encodeTo(w);
+            w.endLenDelim(__mark);
+        }
+        if (value_nullable) {
+            w.writeTag(3, 0);
+            w.writeBool(value_nullable);
+        }
+    }
+}

core/src/main/proto/encodings.proto

@@ -148,6 +148,12 @@ message VariantMetadata {
   optional vortex.dtype.DType shredded_dtype = 1;
 }
 
+message ParquetVariantMetadata {
+  bool has_value = 1;
+  optional vortex.dtype.DType typed_value_dtype = 2;
+  bool value_nullable = 3;
+}
+
 message OnPairMetadata {
   vortex.dtype.PType uncompressed_lengths_ptype = 1;
   uint32 dict_size = 3;

core/src/test/java/io/github/dfa1/vortex/core/model/EditionsTest.java

@@ -96,7 +96,7 @@ void core2026_08_3_isTheFullCoreSet() {
                     EncodingId.FASTLANES_RLE, EncodingId.VORTEX_FIXED_SIZE_LIST,
                     EncodingId.VORTEX_LISTVIEW, EncodingId.VORTEX_MASKED,
                     EncodingId.VORTEX_ONPAIR, EncodingId.VORTEX_MAP,
-                    EncodingId.VORTEX_VARIANT, new EncodingId.Custom("vortex.parquet.variant")));
+                    EncodingId.VORTEX_VARIANT, EncodingId.VORTEX_PARQUET_VARIANT));
         }
 
         @Test

docs/compatibility.md

@@ -38,7 +38,7 @@ only the built-in decoders in `reader`; no encoder class is loaded.
 |------|------------|-------------|
 | `DType::Union` (`fbs.DType.Type.Union = 12`) | Rust 0.71.0 | ❌ Decode throws `VortexException("unsupported DType typeType=12")`. No `DType.Union` variant in Java's sealed type. |
 | `vortex.onpair` experimental string encoding | Rust 0.74.0 | ✅ Read and written. In `core2026.08.1`, so default cascading writes offer it, as Rust's default compressor does; the trained dictionary is valid for Rust but not byte-identical to Rust's (Rust's sampling RNG is not portable). |
-| `vortex.variant` arbitrary nested objects | Rust (`vortex.parquet.variant`) | ⚠️ Java encodes/decodes variant columns of **typed scalar** values (constant / chunked-of-constants core, optional shredded child); Java↔Rust round-trip verified. Arbitrary nested JSON objects and real path-based shredding need the `vortex.parquet.variant` physical encoding — deferred ([ADR 0014](../adr/0014-variant-encoding-strategy.md)). ❌ Reading is a real gap: vortex-jni's default writer turns an Arrow `arrow.parquet.variant` column into `vortex.parquet.variant` (`core2026.08.3`), which vortex-java has no decoder for. |
+| `vortex.variant` arbitrary nested objects | Rust (`vortex.parquet.variant`) | ⚠️ Java reads `vortex.parquet.variant`, the Apache Variant binary encoding vortex-jni's default writer uses for Arrow `arrow.parquet.variant` columns, as a struct of per-row `metadata`/`value` binaries; it does not interpret the Variant binary itself. Java writes variant columns of **typed scalar** values only (constant / chunked-of-constants core, optional shredded child), via `vortex.variant`; writing `vortex.parquet.variant` is not implemented ([ADR 0014](../adr/0014-variant-encoding-strategy.md)). |
 | Arrow extension array import affecting Variant shape | Rust 0.74.0 (#8125) | Untested against the currently pinned v0.85.0 fixtures; #8125 not yet re-verified. |
 | `vortex.dict` **layout** over a values pool that is neither VarBin- nor primitive-shaped (e.g. a dict-encoded `vortex.uuid`, whose storage is `FixedSizeList(U8, 16)`) | Not written by Rust: its dict layout admits only `Primitive \| Utf8 \| Binary` (`dict_layout_supported`), and its dict compressor schemes only integers, floats and strings | ⚠️ Unreachable from Rust- or Java-written files. Every type vortex-jni writes reads back exactly, dict-encoded or not (`DictAllTypesInteropIntegrationTest`, which also fails if a Rust bump starts dict-encoding another type). A foreign file with such a pool fails with `VortexException("unsupported dict values shape: …")`. |
 | Duplicate struct field names | Rust writer rejects ("StructLayout must have unique field names"); Rust reader tolerates foreign files (first-match access) | ⚠️ Deliberate divergence on read: Java rejects such files with `VortexException("duplicate field name in file schema")` instead of tolerating them — the name-keyed `Chunk` API cannot represent both columns, and silent column loss is worse than a loud failure on a file the reference writer refuses to produce. Java's writer mirrors the Rust writer's rejection. |
@@ -122,6 +122,7 @@ decimals ([#430](https://github.com/dfa1/vortex-java/pull/430)) and nulls in nul
 | `fastlanes.rle`             | `RleEncodingDecoder`             | `RleEncodingEncoder`             | ✅      | ✅      | Chunk-based RLE. Integers and floats (Rust's int and float RLE schemes); float runs compare raw bits, so -0.0 and NaN payloads round-trip. Cascades values/indices/offsets |
 | `vortex.patched`            | `PatchedEncodingDecoder`         | `PatchedEncodingEncoder`         | ✅      | ✅      | Primitive PTypes; base + chunked patches (1024-elem blocks)            |
 | `vortex.variant`            | `VariantEncodingDecoder`         | `VariantEncodingEncoder`         | ✅      | ✅      | Canonical container; constant / chunked-of-constants core + optional shredded child. Typed-scalar values only — nested objects need `parquet.variant` (ADR 0014) |
+| `vortex.parquet.variant`    | `ParquetVariantEncodingDecoder`  | —                                | ✅      | ❌      | Read only: decodes to a struct `{metadata, value?, typed_value?}` of Apache Variant binaries (Arrow's `arrow.parquet.variant` storage shape), masked by row validity. Writing is not implemented; Java writes typed-scalar variants through `vortex.variant` (ADR 0014) |
 | `vortex.onpair`             | `OnPairEncodingDecoder`          | `OnPairEncodingEncoder`          | ✅      | ✅      | Utf8, Binary; `core2026.08.1`, a cascade candidate under the default edition (competes with FSST, as in Rust) |
 
 ### Decode shape
@@ -215,7 +216,7 @@ guarantee once frozen (ADR 0023) — a write-time/read-time policy, not part of
 | `core2026.08.0` | `core` | no encoding (Rust adds the `vortex.zoned` layout and zone-map aggregates, not modelled) |
 | `core2026.08.1` | `core` | `vortex.onpair` |
 | `core2026.08.2` | `core` | `vortex.map` |
-| `core2026.08.3` | `core` | `vortex.variant`, `vortex.parquet.variant` ❌ not implemented — **default write target** |
+| `core2026.08.3` | `core` | `vortex.variant`, `vortex.parquet.variant` (read only) — **default write target** |
 | `preview2026.08.0` | `preview` | nothing yet |
 | `zstd2026.02.0` | `zstd` | `vortex.zstd_buffers` ❌ not implemented (buffer-level Zstd for GPU decode; declared by Rust's `vortex-zstd` plugin, opt-in) |
 
@@ -345,5 +346,5 @@ Cross-language round-trips tested against Rust-written fixture files hosted at
 | `clickbench_hits_5k.regular.vortex` | ✅      | Full scan of every Utf8 column (`vortex.onpair`) compared against vortex-jni |
 | `masked.vortex`                     | ❓      | No fixture through v0.86.1, and vortex-jni's writer folds validity into each encoding rather than emitting `vortex.masked` for nullable primitive, string, struct or list input |
 | `patched.vortex`                    | ❓      | No fixture through v0.86.1; `vortex.patched` is in no edition, so vortex-jni's default writer never emits it |
-| `variant.vortex`                    | ❌      | No fixture through v0.86.1, but vortex-jni writes an Arrow `arrow.parquet.variant` column as `vortex.variant` over `vortex.parquet.variant`, which vortex-java cannot decode |
+| `variant.vortex`                    | ✅      | No fixture through v0.86.1; covered instead by `ParquetVariantInteropIntegrationTest`, which has vortex-jni write an Arrow `arrow.parquet.variant` column (`vortex.variant` over `vortex.parquet.variant`, plain and shredded) |
 | `map.vortex`                        | ✅      | New in v0.86.1; also covered directly (both directions, nullable) by the vortex-jni oracle (issue #351) |

docs/reference.md

@@ -347,9 +347,9 @@ targeted edition is ever persisted into a `.vortex` file.
 | `cumulativeMembers(Edition)`         | The edition's own additions plus every earlier same-family edition's         |
 | `owningEdition(EncodingId)`          | The edition an id first joined, or empty if it belongs to none               |
 
-vortex-java implements every `core`-family encoding except `vortex.parquet.variant`, and not `zstd`'s
-`vortex.zstd_buffers`; both resolve to `EncodingId.Custom` and are stored in the catalog anyway,
-mirroring upstream faithfully.
+vortex-java implements every `core`-family encoding (`vortex.parquet.variant` read only), but not
+`zstd`'s `vortex.zstd_buffers`, which resolves to `EncodingId.Custom` and is stored in the catalog
+anyway, mirroring upstream faithfully.
 `fastlanes.delta` and `vortex.patched` belong to no edition, as in Rust.
 
 ### Writer integration (`WriteOptions#editions()`)