test(index): make test_query_delta_indices deterministic#7316
Open
wombatu-kun wants to merge 1 commit into
Open
test(index): make test_query_delta_indices deterministic#7316wombatu-kun wants to merge 1 commit into
wombatu-kun wants to merge 1 commit into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
5731d7c to
c7eab1e
Compare
test_query_delta_indices builds an IVF_HNSW_SQ index over random vectors and asserts the exact top-2 neighbor ids ([0, 1000]). The build was non-deterministic: the input vectors were unseeded, IVF centroid training drew a random training-data sample, and k-means init was unseeded, so the approximate search occasionally dropped the exact match and returned a near neighbor instead (the CI failure showed [116, 1000]). Seed the input vectors and supply fixed IVF centroids for the SQ case via try_with_centroids, which skips IVF centroid training (the random sample and the unseeded k-means init). HNSW level assignment is already seeded and SQ does not sample at this size, so the build is now reproducible. The 2 IVF partitions, nprobes(2), all structural assertions, and the exact [0, 1000] assertion are unchanged. The ivf_pq case is left as-is; it was never the flaky case. Refs lance-format#2125, lance-format#6953. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
c7eab1e to
ef967d5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
test_query_delta_indices(rust/lance/src/index/append.rs) is a recurring flake on its IVF_HNSW_SQ case, previously tracked by #2125 and partially addressed by #6953. It surfaced again as an unrelated CI failure on #7310: https://github.com/lance-format/lance/actions/runs/27660888625/job/81804843234 (assertion left == right failed, left: [116, 1000], right: [0, 1000]).The test writes 1000 random vectors, appends the same vectors with shifted ids (so row
iand rowi+1000share a vector), builds two delta index segments, and asserts the top-2 nearest tovector[0]are exactly[0, 1000](one exact hit per segment). The IVF_HNSW_SQ build is non-deterministic at three points that all feed the trained centroids -> partition split -> per-partition HNSW graph -> whether the approximate search keeps the exact match:generate_random_array;So the search occasionally drops the exact match (id 0) and returns a near neighbor instead.
Fix
Test-only change, no production behavior change:
generate_random_array_with_seed;IvfBuildParams::try_with_centroids, which skips IVF centroid training entirely (both the random sample and the unseeded k-means init).HNSW level assignment is already seeded (
HNSW_LEVEL_RNG_SEED) and SQ does not sample at this size (sample_size256 * 2^8>> 1000), so with seeded data and fixed centroids the whole build is reproducible. The 2 IVF partitions,nprobes(2), all structural assertions, and the exact[0, 1000]assertion are unchanged. Theivf_pqcase is left as-is; it was never the flaky case.Tests
Ran
test_query_delta_indices(both parametrized cases) 300x with zero failures; the SQ case now returns[0, 1000]identically on every run.