Skip to content

HDDS-17778. Drop httpclient and httpcore from the Ozone filesystem jar - #12012

Open
yandrey321 wants to merge 2 commits into
apache:masterfrom
yandrey321:HDDS-17778
Open

yandrey321 wants to merge 2 commits into
apache:masterfrom
yandrey321:HDDS-17778

Conversation

@yandrey321

@yandrey321 yandrey321 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Problem. ozone-common and ozone-filesystem-common declare httpclient and httpcore only for two classes, used in four places:

  • URIBuilder builds the o3fs:// / ofs:// URI in BasicOzoneFileSystem and BasicRootedOzoneFileSystem, and
    the OM DB checkpoint URL in OMNodeDetails;
  • org.apache.http.ParseException is thrown twice in OFSPath.

Those declarations are why the shaded filesystem jar bundles both libraries: 773 entries, about 0.97 MB compressed. The jar is shipped with every job that uses -libjars or spark.jars.

Implementation idea. Use JDK classes instead, then remove the two dependency declarations:

  • Filesystem URIs: new URI(scheme + "://" + authority). That is the string URIBuilder builds and parses, so
    the result is the same, including the URISyntaxException for an illegal authority.
  • Checkpoint URL: the 5-argument URI constructor, with the query built from two constant keys and booleans.
  • OFSPath: throws IllegalArgumentException with the same messages.
  • Guard: a ban-httpcomponents enforcer rule in ozone-filesystem-shaded.

Key decisions.

  • No IPv6 bracketing for the filesystem URIs. URIBuilder brackets an unbracketed IPv6 literal, and the JDK
    parse doesn't. Both file systems reject unbracketed IPv6 authorities earlier in initialize, so the input never
    gets that far.
  • String parse rather than the 5-argument constructor for the filesystem URIs. getAuthority() returns the
    decoded authority. The multi-argument constructors would quote it, so ofs://om%20x/ would be accepted instead of failing with "Invalid Ozone endpoint", and ofs://om%2Fx/ would come out differently.
  • The ban is enabled only in ozone-filesystem-shaded. The unshaded client modules still get httpclient from
    hadoop-common / hadoop-hdfs-client, so a ban there would fail the build. The shaded module catches both ways back: a re-added declaration, and code that uses httpclient, because dependency analysis would then require a
    declaration.

Side effects.

  • OFSPath throws IllegalArgumentException instead of org.apache.http.ParseException. Both are unchecked, and nothing catches the http one. The catch (ParseException ...) blocks in TrashPolicyOzone, S3ConditionalRequest and ReconUtils are java.text.ParseException.
  • Applications that depend on the unshaded ozone-common / ozone-filesystem-common artifacts no longer get httpclient from them; they still get it from Hadoop. All Ozone modules that use httpclient (s3gateway, cli-shell, vapor, integration tests) declare it themselves.
  • No wire, proto, config or on-disk changes.

Co-authored-by: Claude Opus 5.5 noreply@anthropic.com

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-17778

How was this patch tested?

Unit + integration tests.

Jar contents: ozone-filesystem-hadoop3 no longer has org/apache/ozone/shaded/org/apache/http,
mozilla/public-suffix-list.txt or META-INF/maven/org.apache.httpcomponents. Nothing else changed apart from
that. It is now 69.2 MB on disk.

@github-actions github-actions Bot added the ofs label Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant