HDDS-1159. Fix flaky testGetMatchingContainerMultipleThreads - #11048
Merged
Conversation
…nerMultipleThreads
chihsuan
reviewed
Aug 20, 2026
chihsuan
left a comment
chihsuan
left a comment
Contributor
There was a problem hiding this comment.
Thanks @shuan1026 Overall looks good, I left two comments.
| @@ -224,7 +224,7 @@ public void testGetMatchingContainer() throws IOException { | |||
| @Test | |||
| @Flaky("HDDS-1159") | |||
Contributor
There was a problem hiding this comment.
This annotation @Flaky still excludes the test from regular CI. Should we remove it and its unused import?
| return null; | ||
| }, executor); | ||
| } | ||
| CompletableFuture.allOf(futures).join(); |
Contributor
There was a problem hiding this comment.
Could this use a bounded get(timeout, TimeUnit)? join() introduces an unbounded, non-interruptible wait over 100,000 tasks, so a stuck task may hold the test until the Surefire fork timeout.
Contributor
Author
There was a problem hiding this comment.
Thanks for the review! Replaced join() with a bounded get(60, TimeUnit.SECONDS).
Verified with a flaky-test-check run: https://github.com/shuan1026/ozone/actions/runs/32497310074
chungen0126
approved these changes
Aug 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
TestContainerStateManagerIntegration#testGetMatchingContainerMultipleThreadswas tagged@Flaky("HDDS-1159")because it intermittently failed withexpected: <3> but was: <2>(occasionally<1>) at the pipeline container-count assertion.Root cause: the test fires 100,000
CompletableFuture.supplyAsync(...)calls toContainerManager.getMatchingContaineron a 4-thread pool, then immediately asserts that the pipeline already hasnumContainerPerOwnerInPipeline(3) containers without joining those futures. The loop finishing only means the tasks were submitted. When the pool has not yet allocated the 2nd/3rd container, the assertion fails. The existingThread.sleep(5000)runs after the assertion, so it never stabilizes it.This is a test timing issue:
getMatchingContaineralready serializes allocation onpipeline.getId(). Reproduced at 2.5% (5/200) viaflaky-test-check(10×20), matching the historical Jira signature.This PR waits for the work to finish before asserting:
CompletableFuture.allOf(...).join()before checking container count.Thread.sleep(5000)and the unusedInterruptedExceptionon the method.What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-1159
How was this patch tested?
https://github.com/shuan1026/ozone/actions/runs/31984414534
https://github.com/shuan1026/ozone/actions/runs/32152547832