Skip to content

HDDS-17477. Support SHA512 and XXHASH trailing checksums in S3 Gateway - #11980

Draft
rich7420 wants to merge 4 commits into
apache:masterfrom
rich7420:HDDS-17477
Draft

rich7420 wants to merge 4 commits into
apache:masterfrom
rich7420:HDDS-17477

Conversation

@rich7420

@rich7420 rich7420 commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

S3 Gateway now accepts and verifies SHA512, XXHASH64, XXHASH3 and XXHASH128 trailers through the existing streaming checksum path. Hash4j supplies the three XXHASH algorithms without native binaries; its license notices and the distribution jar list are included.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-17477

How was this patch tested?

At cf8c064167:

  • JDK 21: 497 targeted S3 Gateway unit cases and 14 SDK cases passed across standard and Ratis streaming configurations, including signed and unsigned SHA512 uploads.
  • Restoring the two checksum implementation files to the base version caused the 32 new-algorithm endpoint cases to fail with InvalidRequest instead of BadDigest; restoring the fix passed.
  • Checkstyle, PMD, license, RAT on a clean source export, and the distribution dependency check passed.
  • Fork CI passed: 113 jobs succeeded and 2 were skipped. S3 Gateway reported 2,119 cases and S3 integration reported 391, with no failures, errors or skips.

XXHASH coverage uses independent AWS CRT reference values in stream and endpoint tests. SDK coverage uses SHA512 because XXHASH calculation requires the optional native AWS CRT dependency.

Copilot AI balanced review requested due to automatic review settings November 7, 2026 15:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@github-actions github-actions Bot added the s3 S3 Gateway label Nov 7, 2026
@rich7420
rich7420 marked this pull request as draft November 9, 2026 08:54
@rich7420

rich7420 commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Moving this to draft for now. HDDS-17478 moves TrailerHeader into its own class and adds a shared base for both chunk streams, so I'll update this once that's in.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

s3 S3 Gateway

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants