Skip to content

[kafka] Transcode mapped raw and string records to Arrow #4282

Description

@gyang94

Purpose

Part of #4185. PR06 of the Kafka compatibility sequence.

Use the DDL mapping contract to decode raw/string keys and values, assemble physical rows with optional timestamps and ordered headers, and encode owned native Arrow log bytes.

Acceptance criteria

  • Raw bytes and strict UTF-8 strings map to the configured physical columns.
  • Nullable values, empty bytes, NOT NULL violations and duplicate/nullable headers are handled correctly.
  • The native batch uses the table schema ID and remains readable after Arrow resources close.
  • Arrow writers and allocators close on encoding failures.

Dependencies

PR05 Produce protocol (#4281); reuses the PR03 DDL contract (#4277 / #4275).

Scope boundary

No JSON decoding, gateway append, protocol registration or core module changes.

Implementation

Draft PR #4285: #4285

Java 11 validation: 75 unit tests and 2 integration tests passed at 503136295d6bbaed81b7e7845dceb2d64db2d9d4. Review and merge remain pending.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions