Repository navigation
HDDS-17738. Fall back to unordered snapdiff report when the dependency graph has a cycle. - #12013
SaketaChalamchala wants to merge 1 commit into
Conversation
smengcl
left a comment
There was a problem hiding this comment.
Thanks @SaketaChalamchala for the patch
| assertEquals("Report is dependency unordered.", | ||
| "Report is dependency " + (store.isReportDependencyOrdered() ? "ordered" : "unordered") + "."); |
There was a problem hiding this comment.
This one builds the reason string inside the test, so it passes whenever the assertFalse above passes and does not touch SnapshotDiffManager. Could we drop it? assertFalse(store.isReportDependencyOrdered()) already checks the outcome, and the reason text itself comes from updateJobStatusToDone.
| for (int i = 0; i < entries.size(); i++) { | ||
| assertArrayEquals(explicitUnordered.get(i).getSourcePath(), entries.get(i).getSourcePath()); | ||
| assertArrayEquals(explicitUnordered.get(i).getTargetPath(), entries.get(i).getTargetPath()); | ||
| } |
There was a problem hiding this comment.
nit: the path checks above already pin the order (a -> b, then b -> a). If we keep the comparison with the unordered run, assertEquals(explicitUnordered, entries) might be simpler, since DiffReportEntry implements equals over the type and both paths. It would also catch a size mismatch, which this loop over entries does not. The four put calls repeated for unorderedStore could go into a small helper shared by both stores.
Found by TLA+ model checking and code review of the OM optimized snapshot diff paths under HDDS-15926, on commit d3b75c7.
The patch is generated with Specula (Claude Opus 5.5).
What changes were proposed in this pull request?
For optimized FSO snapshot diff with dependency ordering enabled, a cyclic dependency graph (e.g. two bucket-root files swapping names) used to fail report generation with
IOException(Dependency graph cycle for job…). The job endedFAILED, and each subsequent ordered request for the same snapshot pair recomputed the diff and failed again. Unordered requests for the same pair could still complete.This change aligns cycle handling with the existing in-memory dependency node limit fallback: when topological sort detects a cycle,
MergeJoinSnapDiffWriterwrites the report viaflushUnorderedDependencyNodesin dependency-node index order, returns a successful write withisReportDependencyOrdered() == false, andSnapshotDiffManagermarks the jobDONEwith reason "Report is dependency unordered." The job’s ordered-request flag is unchanged so a later ordered request can reuse the completed job instead of recomputing indefinitely.Implementation (OM /
MergeJoinSnapDiffWriter):buildOrderedNodeIdsreturns null on a dependency-graph cycle (WARNlog) instead of throwing.orderAndWriteReportreturnsboolean, flushes unordered nodes onnull, and propagates the outcome throughdependencyOrderingEnabledRefin writeReport.IllegalStateExceptions(non-cycle) are still propagated.What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-17738
How was this patch tested?
Unit Test added:
testDependencyOrderingFallsBackOnCycleinTestMergeJoinSnapDiffWriter(swap-rename scenario), asserting successful write, unordered outcome /DONE-style reason,RENAME a→bandRENAME b→a, same row order as explicit unorderedwriteReport, and noputOrderedReportEntries.