HDDS-15014. Design doc for Speed up EC container decommission - #10086
Draft
jojochuang wants to merge 3 commits into
Draft
HDDS-15014. Design doc for Speed up EC container decommission#10086jojochuang wants to merge 3 commits into
jojochuang wants to merge 3 commits into
Conversation
errose28
self-requested a review
April 20, 2026 16:11
errose28
added
design
and removed
documentation
jojochuang
force-pushed
the
HDDS-15014
branch
2 times, most recently
from
April 21, 2026 22:30
13169f7 to
4fc087b
Compare
smengcl
reviewed
May 28, 2026
| We will introduce new configuration properties in `ReplicationManagerConfiguration` to control the behavior and protect cluster resources. | ||
|
|
||
| 1. **New Configuration Keys:** | ||
| * `hdds.scm.replication.decommission.ec.reconstruction.enabled` (Boolean, default: false): Feature flag to enable/disable the switch to reconstruction during decommission. |
Contributor
There was a problem hiding this comment.
Which DN will act as the client when performing EC reconstruction?
The source DN that has one of the stripes? Or the target DN that will receive the reconstructed block
|
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days. |
…tus. Reflect merged Phase 3 (HDDS-15412), unmerged Phases 1-2, superseded HDDS-15073/15074, HDDS-15327 prerequisite, and Solution 2 deprecation. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: If9298f6bc67f09bf793104d6225641ddb1242365
Contributor
Author
|
Updated the design doc on branch
|
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new design document (HDDS-15014) describing how Apache Ozone can speed up EC container decommission by transitioning from single-source EC replica copy to multi-source EC reconstruction, including a phased rollout plan across SCM and DataNode components.
Changes:
- Introduces a design doc summarizing current EC decommission bottlenecks and congestion considerations (node-level and disk-level).
- Documents a phased implementation plan (SCM config + global reconstruction cap, dynamic replication→reconstruction switching, per-volume fairness, observability).
- Captures requirements/non-goals and an expected-results section for the proposed approach.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
|
||
| # Problem: | ||
|
|
||
| Decommissioning of dense datanode is, especially the datanodes of mostly EC containers, is very slow: |
| [https://ozone.apache.org/docs/next/administrator-guide/operations/node-decommissioning-and-maintenance/datanodes/datanode-decommission](https://ozone.apache.org/docs/next/administrator-guide/operations/node-decommissioning-and-maintenance/datanodes/datanode-decommission) | ||
|
|
||
| ``` | ||
| When we initiate the process of decommissioning, first we check the current state of the node, ideally it should be IN_SERVICE, then we change it's state to DECOMMISSIONING and start the process of decommissioning, it goes through a workflow where the following happens: |
|
|
||
| # Disk-level congestion | ||
|
|
||
| In fact, because Ozone replication manager does not control I/O at disk level (only node level), multiple tasks may land at the same disk at the same time, and the single disk becomes the bottleneck for the entire datanode, and thus the enter cluster. |
| # Expected result: | ||
|
|
||
| 1. Increase the parallelism of transfer to meet SLAs. | ||
| 2. transition from a "single-source replication" model to a "multi-source reconstruction" |
| * Threshold: load factor relative to effective replication limit (Phase 2); RS(3,2) vs RS(6,3) differentiated thresholds remain **future work**. | ||
| * **Disk-Aware Scheduling:** The Datanode (DN) ensures push replication tasks do not overload a single physical disk. HDDS-15412 implements this at the DN via per-volume replication thread pools (no SCM disk-level status reporting). EC reconstruction and reconciliation remain on the global pool. | ||
| * **Bisectional Bandwidth Management:** For RS(6,3), you need to read 6x the data. The specification needs a plan for how the RM selects "source" nodes for reconstruction to avoid creating new bottlenecks in other parts of the cluster. Phase 2 source offloading addresses the decommissioning node; rack-aware source selection is **future work**. | ||
| * **Memory Overhead:** increasing parallelism and adding more re-construction tasks could increase memory overhead. That is expected, but it shouldn’t consume so much it becomes infeasible to deploy. Say, ideally contain the datanode heap to under 31GB, and total process memory (including direct memory, native memory) should not exceed 64GB at any point in time. |
Note that per-volume thread pools (HDDS-15412) replace the outbound-counter and lookahead-dispatch approach; update Phase 1 PR link to apache#11054. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I5ce95991acd7c4d6854256fc9dd4365a91fd3143
Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: Ieb2ee0e9a20249a4d8d5e7434a5ce28684b0c362
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
HDDS-15014. Design doc for Speed up EC container decommission
Please describe your PR in detail:
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15014
How was this patch tested?