Skip to content

HDDS-16238. Latency metrics for internal OM requests are not populated - #11072

Open
errose28 wants to merge 5 commits into
apache:masterfrom
errose28:worktree/purgekey-metrics
Open

HDDS-16238. Latency metrics for internal OM requests are not populated#11072
errose28 wants to merge 5 commits into
apache:masterfrom
errose28:worktree/purgekey-metrics

Conversation

@errose28

@errose28 errose28 commented Aug 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

The OM's ProtocolMessageMetrics is only used in the server side translator. This captures end-to-end latency for all client facing requests (like RenameKey), but not for internal OM requests submitted directly to Ratis by background services (like PurgeKeys). This causes latency metrics for internal requests to always be 0.

This PR wires the metrics to OzoneManagerRatisUtils#submitRequest as well to capture latency of requests submitted directly to the Ratis server.

What is the link to the Apache JIRA

HDDS-16238

How was this patch tested?

  • Unit tests added for OzoneManagerRatisUtils and all existing background services.
  • Green CI on my fork.

@errose28

errose28 commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

cc @ptlrs @yandrey321

yandrey321
yandrey321 approved these changes Aug 20, 2026
View reviewed changes

@yandrey321 yandrey321 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@errose28
errose28 marked this pull request as ready for review August 21, 2026 15:28
rich7420
rich7420 approved these changes Aug 21, 2026
View reviewed changes

@rich7420 rich7420 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
it's a clean one
thanks for the patch @errose28

@github-actions

github-actions Bot commented Sep 12, 2026

Copy link
Copy Markdown

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

@github-actions github-actions Bot added the stale label Sep 12, 2026
@errose28 errose28 removed the stale label Sep 16, 2026
@github-actions github-actions Bot added the om label Sep 23, 2026
@github-actions

github-actions Bot commented Oct 15, 2026

Copy link
Copy Markdown

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

@github-actions github-actions Bot added the stale label Oct 15, 2026
@yandrey321

yandrey321 commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

@errose28 can we resurrect this PR?

@github-actions github-actions Bot removed the stale label Oct 16, 2026
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

@github-actions github-actions Bot added the stale label Nov 7, 2026
@yandrey321

yandrey321 commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

@errose28 could you please rebase the branch? Its a good change and we should get it merged.

@github-actions github-actions Bot removed the stale label Nov 8, 2026
@errose28
Merge branch 'master' into worktree/purgekey-metrics
f8adeeb 
* master: (892 commits)
  HDDS-16784. Run rebot in pod in kubernetes check (apache#11938)
  HDDS-17705. Stop Recon OM sync before closing the metadata manager (apache#11936)
  HDDS-17755. Use isSupportedBy() for OzoneManagerVersion check (apache#12002)
  HDDS-17770. Bump zizmorcore/zizmor-action to 0.8.1 (apache#11987)
  HDDS-16701. S3G: Avoid calling getS3Volume in DELETE_KEY (apache#11740)
  HDDS-16616. Replace Pair/Triple with domain objects in SCM node decommission code (apache#11984)
  HDDS-17500. Return x-amz-server-side-encryption on GET and HEAD for encrypted objects (apache#11972)
  HDDS-17478. Share aws-chunked parsing between SignedChunksInputStream and UnsignedChunksInputStream (apache#11992)
  HDDS-17410. Record STS revocation cutoff when revoking tenant access id (apache#11796)
  HDDS-17635. Datanode ReadBlock: a cancelled streaming call is not a container failure (apache#11891)
  HDDS-17706. Avoid retrying datanode usage query for unknown UUID (apache#11947)
  HDDS-17407. Allow for clock skew with revocation (apache#11952)
  HDDS-17421. SCM terminates when it applies Raft log entry written by pre-ZDU SCM finalization (apache#11925)
  HDDS-16951. Remove the per-call InfoBucket from rooted-ofs rename on FSO buckets (apache#11674)
  HDDS-17602. Use different runners for some jobs (apache#11977)
  HDDS-17409. Record STS revocation cutoff when revoking S3 secrets (apache#11794)
  HDDS-17515. Move KeyLifecycleService properties to a reconfigurable config group (apache#11912)
  HDDS-17465. Recon keeps the old container mappings of an overwritten key (apache#11762)
  HDDS-17503. [Recon] Report OM pending deletion as purgeable and snapshot-held size (apache#11855)
  HDDS-17581. Add TracingUtil span helpers and span-kind aware tracing proxies (apache#11852)
  ...

Conflicts:
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestTrashOzoneFileSystem.java
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/service/TestDirectoryDeletingService.java
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/service/TestQuotaRepairTask.java
@errose28

errose28 commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

Just did the reverse merge from master. I skimmed it and it looks correct, although it's been a while since I looked at this. If more updates are needed I may not have time to get to them until the remaining ZDU bugs are resolved.

ptlrs
ptlrs reviewed Nov 10, 2026
View reviewed changes

@ptlrs ptlrs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @errose28.

I had taken a look at this PR a while back and at that time one my concerns was that some of the existing metrics/dashboards may break. From the user's perspective, it may appear that the number of operations like deletion are now higher post this PR than what they may have triggered themselves.

It would be good to have some data about what the difference would look like and if it is acceptable. Otherwise system generated operations should be tagged in a way that that they can be isolated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

3 more reviewers
@ptlrs ptlrs ptlrs left review comments
@rich7420 rich7420 rich7420 approved these changes
@yandrey321 yandrey321 yandrey321 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

om

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@errose28 @yandrey321 @ptlrs @rich7420