Skip to content

test(index): make test_query_delta_indices deterministic#7316

Open
wombatu-kun wants to merge 1 commit into
lance-format:mainfrom
wombatu-kun:fix/flaky-test-query-delta-indices
Open

test(index): make test_query_delta_indices deterministic#7316
wombatu-kun wants to merge 1 commit into
lance-format:mainfrom
wombatu-kun:fix/flaky-test-query-delta-indices

Conversation

@wombatu-kun

Copy link
Copy Markdown
Contributor

Problem

test_query_delta_indices (rust/lance/src/index/append.rs) is a recurring flake on its IVF_HNSW_SQ case, previously tracked by #2125 and partially addressed by #6953. It surfaced again as an unrelated CI failure on #7310: https://github.com/lance-format/lance/actions/runs/27660888625/job/81804843234 (assertion left == right failed, left: [116, 1000], right: [0, 1000]).

The test writes 1000 random vectors, appends the same vectors with shifted ids (so row i and row i+1000 share a vector), builds two delta index segments, and asserts the top-2 nearest to vector[0] are exactly [0, 1000] (one exact hit per segment). The IVF_HNSW_SQ build is non-deterministic at three points that all feed the trained centroids -> partition split -> per-partition HNSW graph -> whether the approximate search keeps the exact match:

  • input vectors come from the unseeded generate_random_array;
  • IVF centroid training samples a random 512-of-1000 subset with unseeded RNGs;
  • k-means centroid init is unseeded.

So the search occasionally drops the exact match (id 0) and returns a near neighbor instead.

Fix

Test-only change, no production behavior change:

  • seed the input vectors with generate_random_array_with_seed;
  • supply fixed IVF centroids for the SQ case via IvfBuildParams::try_with_centroids, which skips IVF centroid training entirely (both the random sample and the unseeded k-means init).

HNSW level assignment is already seeded (HNSW_LEVEL_RNG_SEED) and SQ does not sample at this size (sample_size 256 * 2^8 >> 1000), so with seeded data and fixed centroids the whole build is reproducible. The 2 IVF partitions, nprobes(2), all structural assertions, and the exact [0, 1000] assertion are unchanged. The ivf_pq case is left as-is; it was never the flaky case.

Tests

Ran test_query_delta_indices (both parametrized cases) 300x with zero failures; the SQ case now returns [0, 1000] identically on every run.

@github-actions github-actions Bot added the chore label Jun 17, 2026
@codecov

codecov Bot commented Jun 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@wombatu-kun wombatu-kun force-pushed the fix/flaky-test-query-delta-indices branch from 5731d7c to c7eab1e Compare June 18, 2026 05:54
test_query_delta_indices builds an IVF_HNSW_SQ index over random vectors and asserts the exact top-2 neighbor ids ([0, 1000]). The build was non-deterministic: the input vectors were unseeded, IVF centroid training drew a random training-data sample, and k-means init was unseeded, so the approximate search occasionally dropped the exact match and returned a near neighbor instead (the CI failure showed [116, 1000]).

Seed the input vectors and supply fixed IVF centroids for the SQ case via try_with_centroids, which skips IVF centroid training (the random sample and the unseeded k-means init). HNSW level assignment is already seeded and SQ does not sample at this size, so the build is now reproducible. The 2 IVF partitions, nprobes(2), all structural assertions, and the exact [0, 1000] assertion are unchanged. The ivf_pq case is left as-is; it was never the flaky case.

Refs lance-format#2125, lance-format#6953.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wombatu-kun wombatu-kun force-pushed the fix/flaky-test-query-delta-indices branch from c7eab1e to ef967d5 Compare June 18, 2026 14:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant