Repository navigation
fix: F16 columns (constant, dict, rle, sparse encodings; statistics Rust can read) - #517
Merged
Merged
Conversation
ConstantEncodingEncoder#isConstant and #readFirstRaw each read element 0 (and, for floats, converted it to raw bits) again; constantBits does both in one pass, with the first value in a local outside the loop and the empty array an explicit early return. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sko6B2519bE1TkpPx7WKxf
ConstantEncodingEncoder and ConstantEncodingDecoder had no F16 case, so a constant F16 column failed to write. The scalar is Rust's f16_value (the half bits as a u64); the decoder also accepts the legacy uint64_value form. The ptype switches lose their default branch, so a future PType fails to compile instead of at write time. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sko6B2519bE1TkpPx7WKxf
DictEncodingDecoder, DictLayoutDecoder and RleEncodingDecoder had no F16 case although the cascade writes both encodings for F16, so such files could not be read back; SparseEncodingEncoder and SparseEncodingDecoder had none at all. New DictFloat16Array, LazyRleFloat16Array and LazySparseFloat16Array keep the 16 raw bits (a NaN payload or -0.0 survives materialize) and ignore the pool's widened floats. Float16EncodingsRoundTripTest writes eight data shapes under the default cascade and each of the four encodings forced, and compares raw bits. The Rust interop test for F16 stays disabled (vortex-jni exports no F16). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sko6B2519bE1TkpPx7WKxf
PrimitiveEncodingEncoder wrote an F16 column's array-stat min/max as f32 scalars, which Rust rejects (expected F32 dtype for F32Value, got f16), so vortex-jni could not read any F16 file Java wrote. The scalars are now f16_value, as Rust types them, and the zone-map codec and the reader's ArrayStats read that form. Float16InteropIntegrationTest checks both directions against vortex-jni (Java writes/JNI reads with the default cascade and each of constant, dict, rle and sparse forced; JNI writes/Java reads), comparing raw half bits; the old disabled javaWriter_jniReader_f16_nanAndInf test passes and is enabled. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sko6B2519bE1TkpPx7WKxf
dfa1
force-pushed
the
fix/515-f16-encodings
branch
from
October 10, 2026 14:43
4f5c871 to
ce8dea7
Compare
dfa1
changed the title
This was referenced Oct 10, 2026
dfa1
added a commit
that referenced
this pull request
Oct 10, 2026
Eleven one-per-fix lines (#515, #517, #521) become a single F16 entry that lists the commits; the Calcite F32 fix stays separate, being a different bug. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sko6B2519bE1TkpPx7WKxf
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #515. Four commits, one per change, to be rebase-merged (not squashed):
a4fba444refactor:ConstantEncodingEncoder#constantBitsreplacesisConstant+readFirstRaw(first value read once, empty array explicit).21ab5f5efix:vortex.constanthandles F16 (encoder + decoder;LazyConstantFloat16Array; thePTypeswitches lose theirdefault).198d4607fix:vortex.dict(reader rejected what the cascade wrote),fastlanes.rle(decoder) andvortex.sparse(encoder + decoder) handle F16 (DictFloat16Array,LazyRleFloat16Array,LazySparseFloat16Array).ce8dea70fix: F16 min/max statistics aref16_valuescalars. They weref32, which Rust rejects (expected F32 dtype for F32Value, got f16), so vortex-jni could not read any F16 file Java wrote. AddsFloat16InteropIntegrationTest(28 cases against vortex-jni, both directions) and re-enablesjavaWriter_jniReader_f16_nanAndInf.Verified locally:
./mvnw verify -pl integration -am(397 tests, 2 skipped) and./mvnw test.🤖 Generated with Claude Code