Skip to content

HDDS-17738. Fall back to unordered snapdiff report when the dependency graph has a cycle. - #12013

Draft
SaketaChalamchala wants to merge 1 commit into
apache:masterfrom
SaketaChalamchala:HDDS-17738
Draft

SaketaChalamchala wants to merge 1 commit into
apache:masterfrom
SaketaChalamchala:HDDS-17738

Conversation

@SaketaChalamchala

Copy link
Copy Markdown
Contributor

Found by TLA+ model checking and code review of the OM optimized snapshot diff paths under HDDS-15926, on commit d3b75c7.
The patch is generated with Specula (Claude Opus 5.5).

What changes were proposed in this pull request?

For optimized FSO snapshot diff with dependency ordering enabled, a cyclic dependency graph (e.g. two bucket-root files swapping names) used to fail report generation with IOException (Dependency graph cycle for job …). The job ended FAILED, and each subsequent ordered request for the same snapshot pair recomputed the diff and failed again. Unordered requests for the same pair could still complete.

This change aligns cycle handling with the existing in-memory dependency node limit fallback: when topological sort detects a cycle, MergeJoinSnapDiffWriter writes the report via flushUnorderedDependencyNodes in dependency-node index order, returns a successful write with isReportDependencyOrdered() == false, and SnapshotDiffManager marks the job DONE with reason "Report is dependency unordered." The job’s ordered-request flag is unchanged so a later ordered request can reuse the completed job instead of recomputing indefinitely.

Implementation (OM / MergeJoinSnapDiffWriter):

  • buildOrderedNodeIds returns null on a dependency-graph cycle (WARN log) instead of throwing.
  • orderAndWriteReport returns boolean, flushes unordered nodes on null, and propagates the outcome through dependencyOrderingEnabledRef in writeReport.
  • Other graph IllegalStateExceptions (non-cycle) are still propagated.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-17738

How was this patch tested?

Unit Test added:

  • testDependencyOrderingFallsBackOnCycle in TestMergeJoinSnapDiffWriter (swap-rename scenario), asserting successful write, unordered outcome / DONE-style reason, RENAME a→b and RENAME b→a, same row order as explicit unordered writeReport, and no putOrderedReportEntries.

@smengcl smengcl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @SaketaChalamchala for the patch

Comment on lines +338 to +339
assertEquals("Report is dependency unordered.",
"Report is dependency " + (store.isReportDependencyOrdered() ? "ordered" : "unordered") + ".");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one builds the reason string inside the test, so it passes whenever the assertFalse above passes and does not touch SnapshotDiffManager. Could we drop it? assertFalse(store.isReportDependencyOrdered()) already checks the outcome, and the reason text itself comes from updateJobStatusToDone.

Comment on lines +356 to +359
for (int i = 0; i < entries.size(); i++) {
assertArrayEquals(explicitUnordered.get(i).getSourcePath(), entries.get(i).getSourcePath());
assertArrayEquals(explicitUnordered.get(i).getTargetPath(), entries.get(i).getTargetPath());
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the path checks above already pin the order (a -> b, then b -> a). If we keep the comparison with the unordered run, assertEquals(explicitUnordered, entries) might be simpler, since DiffReportEntry implements equals over the type and both paths. It would also catch a size mismatch, which this loop over entries does not. The four put calls repeated for unorderedStore could go into a small helper shared by both stores.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI-gen om snapshot https://issues.apache.org/jira/browse/HDDS-6517

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants