[kafka] Define DDL table mapping for Kafka compatibility

[gyang94]

Purpose

Closes #4275. Part of #4185.

Define the shared DDL table mapping contract before Metadata and Produce, so both APIs can agree on topic identity, supported tables, and physical record fields. This is PR03 in the foundation sequence, based directly on PR02 (#4260, tracked by #4265); Metadata follows as PR04 (#4261 / #4266).

Brief change log

• Extract topic/table name and ID mapping, including Kafka-compatible topic names and stable IDs over a table's lifetime.
• Validate non-partitioned Arrow log tables without primary keys and resolve kafka.* custom properties for raw/string key and value formats.
• Resolve key/value field projections and optional timestamp/headers columns; reject invalid types, missing fields, overlapping projections, and unsupported mapping options.
• Add mapping regression tests and a native Fluss create-table → metadata-readback integration test. Reuse the existing curator-test version in test scope for the cluster fixture.

Relative to PR02: 10 files, +1249 / -0 lines (769 production/build and 480 test lines). This keeps the complete table contract and its validation together; record decoding, Arrow encoding, Metadata handlers, Produce, JSON, and authentication remain separate PRs.

Tests

Validated on Java 11 at commit 6b9626d16d84b47e632ce9ab43f13af3fce116c1:

• mvn -o -pl fluss-kafka -am install -DskipTests to build/install matching dependencies.
• mvn -o -pl fluss-kafka clean verify: 47 unit tests and 2 integration tests passed, including the native DDL round trip.
• Checkstyle, Spotless, RAT, and git diff --check passed.

The full repository test suite and a Flink SQL runtime were not run. The later delivery stage will cover the full DDL → Metadata → Kafka producer → Fluss readback workflow.

API and Format

Uses the existing table schema and custom-property storage. kafka.value.format is required; key mapping is optional. Raw and string map to one BYTES or STRING column respectively. Configured key fields require kafka.value.fields-include=EXCEPT_KEY; timestamp and headers use explicitly named physical columns. There is no mandatory four-column envelope.

No Kafka API is newly advertised by this PR. Table creation continues through existing Fluss DDL/Admin or the Flink catalog; Kafka CreateTopics/DeleteTopics and automatic creation are out of scope.

Documentation

The contract is described in this PR, source Javadoc and executable tests. User-facing delivery examples follow with the later Produce capability.

Stack and review

1. [kafka] Add request dispatch and transport framework #4259 — PR01: request and transport framework.
2. [kafka] Serve ApiVersions from registered capabilities #4260 — PR02: ApiVersions.
3. This PR — PR03: DDL table mapping ([kafka] Define DDL table mapping for Kafka compatibility #4275).
4. [kafka] Implement Metadata API over Fluss table metadata #4261 — PR04: Metadata ([kafka] Implement Metadata API for pre-created Fluss tables #4266).

This PR remains a draft while its prerequisites are unmerged. It targets Apache main, so the Files changed view includes the foundation prerequisites. Review DDL mapping alone using the incremental comparison against PR02.

Generative AI disclosure: Codex assisted with implementation, testing, and commit organization. Human review is required before merge.