Skip to content

[kafka] Add request dispatch and transport framework - #4259

Open
gyang94 wants to merge 1 commit into
apache:mainfrom
gyang94:codex/kafka-framework
Open

[kafka] Add request dispatch and transport framework#4259
gyang94 wants to merge 1 commit into
apache:mainfrom
gyang94:codex/kafka-framework

Conversation

@gyang94

@gyang94 gyang94 commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Purpose

Closes #4264.

Part of umbrella #4185; this PR does not close the umbrella.

This is the first foundation stage under #4185: request/transport framework, followed by ApiVersions and Metadata. Each stage is tracked by a separate child issue. This PR excludes CreateTopics/DeleteTopics.

Brief change log

  • Introduce request context, typed API specifications and handlers, a frozen capability registry, asynchronous dispatch, and common error mapping.
  • Propagate the listener context through the Kafka channel and recognize configured protocol endpoints in NettyServer.
  • Make request buffer release idempotent and release the response buffer when serialization fails.
  • Add isolated dispatcher, registry, decoder, and buffer-ownership regression tests.
  • Preserve main's existing request-handler implementation in this stage. The ApiVersions follow-up wires the new dispatcher into production.

Scope: 16 files, +1039 / -41 lines including tests. This PR adds no concrete API handler, Produce implementation, topic administration, dependency, or storage format.

Tests

  • mvn -o -pl fluss-rpc,fluss-kafka verify
  • Kafka: 17 unit tests and 1 integration test.
  • RPC: full module unit/integration suites, including NettyServer and authentication coverage; 2 existing tests are skipped.
  • git diff --check; git range-diff confirms unchanged patches after rebasing onto main (b9a1653fa).

Dependencies were rebuilt with mvn -o -pl fluss-kafka -am install -DskipTests on Java 11. This also passed the reactor's Checkstyle, Spotless and RAT checks. The entire repository test suite was not run.

API and Format

Adds internal Kafka extension points; existing Fluss public APIs and storage formats are unchanged. Concrete ApiVersions and Metadata behavior is introduced only in the two follow-ups. Tables remain managed by Fluss DDL.

Documentation

This PR description documents the stage boundary and review order. No end-user Produce or schema-format documentation is introduced here.

Stack

  1. Request/transport framework — this PR.
  2. ApiVersions — [kafka] Serve ApiVersions from registered capabilities #4260.
  3. Metadata — [kafka] Implement Metadata API over Fluss table metadata #4261.

Generative AI disclosure: Codex assisted with implementation, testing, and commit organization. Human review is required before merge.

Introduce API registration, request context, version validation, and
asynchronous error mapping. Fix request buffer ownership and response
serialization cleanup while preserving the existing ApiVersions entry point.

Validated with mvn -o -pl fluss-rpc,fluss-kafka verify.

Co-Authored-By: Codex <noreply@openai.com>
AI-Model: gpt-6
AI-Contributed/Feature: 555/555
AI-Contributed/UT: 525/525
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[kafka] Add request and transport framework

1 participant