Skip to content

HDDS-17205. OM and SCM HA use reverse DNS names instead of configured IP literals for Ratis peer addresses - #11656

Draft
smengcl wants to merge 6 commits into
apache:masterfrom
smengcl:HDDS-17205
Draft

smengcl wants to merge 6 commits into
apache:masterfrom
smengcl:HDDS-17205

Conversation

@smengcl

@smengcl smengcl commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Generated-by: Claude Code (Opus 5.5)

What changes were proposed in this pull request?

When OM or SCM HA nodes are configured with IP literals (for example on an IPv6 only cluster), NodeDetails builds the Ratis peer address from InetSocketAddress.getHostName(), which does a reverse DNS lookup and replaces the literal with whatever name the resolver returns. Several call sites in OMHANodeDetails, OMNodeDetails and SCMHANodeDetails also call getHostName() before the node details are built, and the lookup result is cached in the InetAddress, so even getHostString() returns the resolved name afterwards.

In a Docker compose cluster with static IPv6 addresses this produces peer addresses like ozone-ha-scm1-1.ozone-ha_default:9894, which gRPC rejects because of the underscore, so the Ratis rings do not form. SCM also fails to match the leader in notifyInstallSnapshotFromLeader and resolveTargetPeerId when the stored address differs from the computed one.

This PR:

  • keeps the configured host string in NodeDetails and replaces the remaining getHostName() calls on the OM and SCM HA address paths with getHostString(), so the configured literal is used as is;
  • adds NodeDetails.isRatisAddress(String), which matches the exact Ratis host and port, or any host that resolves to the node's IP with the Ratis port, so clusters whose Ratis configuration already stores host names keep matching;
  • uses isRatisAddress for SCM leader matching in SCMStateMachine and SCMClientProtocolServer.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-17205

How was this patch tested?

  • New unit tests in TestOMNodeDetails and TestSCMConfiguration (fail without the fix), plus existing OM and SCM HA unit tests.
  • Manually, with the ozone-ha compose cluster on an IPv6 only network with static IP literals in the OM and SCM addresses: SCM and OM Ratis rings form on the literal addresses, ozone admin scm transfer works, and the HA acceptance suites pass.

🤖 Generated with Claude Code

… IP literals for Ratis peer addresses

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings October 21, 2026 04:37
@smengcl smengcl added the AI-gen label Oct 21, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Retain both the IP-literal Ratis address regression test and the IPv6 SCM peer-address cases while preserving the production changes from both branches.

Co-authored-by: Codex (GPT-6)
Change-Id: I625e9ddeb2280057d5e4147269ea69dd4d9c7b08

@jojochuang jojochuang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One remaining path can still replace configured IP literals with reverse DNS names before NodeDetails is constructed.

…hecks

OzoneNetUtils.isAddressHostNameLocal and getAddressWithHostNameLocal called getHostName() on the configured address, which caches the reverse DNS name in it. With ozone.network.flexible.fqdn.resolution.enabled, OM and SCM run these checks before building node details, so the Ratis peer address still used the reverse DNS name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jojochuang

Copy link
Copy Markdown
Contributor

Non-blocking, nice-to-have, and unrelated to this PR's fix: could we consider validating rpcAddress in the constructor that accepts an InetSocketAddress?

If null is passed to that constructor, hostAddress remains unset, so an incomplete node definition fails later during address construction. An explicit Objects.requireNonNull(rpcAddress, "rpcAddress == null") would give a clearer error at the source, provided no callers intentionally rely on that case.

This would be specific to the socket-address constructor; the separate host-string constructor intentionally supports lazy initialization of rpcAddress. This can be handled separately and does not block this PR.

@jojochuang jojochuang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still don't quite get it. Was it not an issue for IPv4? Or perhaps because we never used ip literals to test clusters before?

@smengcl

smengcl commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

Was it not an issue for IPv4? Or perhaps because we never used ip literals to test clusters before?

It is not IPv6 specific. An IPv4 literal goes through the same code. It went unnoticed because our test clusters (compose, MiniOzoneCluster, acceptance) configure OM and SCM addresses by host name. For a configured name, getHostName() returns that name and does no lookup, so nothing changes.

It only matters when the address is configured as an IP literal. Even then it often still works: if every host's reverse lookup returns the same name and that name resolves, the ring forms on the name instead of the literal, and nobody notices. It breaks when hosts get different names for the same IP (a short name from /etc/hosts on the node itself, an FQDN from DNS on the others), or when the name can't be used. In compose the name is ozone-ha-scm1-1.ozone-ha_default, and gRPC rejects it because of the underscore.

We hit it first on IPv6 because there the literal is the natural choice: the IPv6 compose cluster uses static addresses and configures OM and SCM by literal.

@smengcl
smengcl requested a review from jojochuang November 6, 2026 22:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants