[TamimEhsan]

Summary

This PR adds an embedded client for RaBitQ-Library (rabitqlib) and its first index type, HNSW-RaBitQ. HNSW-RaBitQ searches an HNSW graph over RaBitQ-quantized codes. The library runs in-process, like the existing zvec and qdrantlocal clients.

pip install vectordb-bench[rabitqlib]
vectordbbench rabitqlibhnsw --path /data/rabitqlib --nbits 9 --m 32 --ef-search 256 \
  --case-type Performance768D1M

How it fits the benchmark's process model

VectorDBBench loads, optimizes and searches in separate processes. An in-process library therefore has to keep all its state in files under --path:

• Load: insert_embeddings appends each batch to a raw buffer (buffer.vectors.f32 and buffer.ids.i64). The two files stay row-aligned because appends hold a lock, which matters because up to 4 insert threads share one client.

• Optimize: the whole buffer goes into one HnswIndex.add. The library trains its centroids inside that call, over the full dataset. The client then saves three things:

  • the index;
  • a map from library point IDs to dataset IDs;
  • a manifest of the build parameters.

  The manifest is written last, so its presence marks a complete build. All three files are named after the index type (rabitqlib_hnsw.*), so later index types can share a path.

• Search: each search process loads the saved index when it unpickles the client (__setstate__). This happens before the concurrent runner's start barrier. Loading in init() would happen after the barrier and count the load time as search time. On GIST1M at concurrency 32, that made reported QPS 23K when the real figure was 55K.

Behaviour and limits

• Static, unfiltered performance cases only. The index can't filter on scalars, so supported_filter_types = [NonFilter] and filtered cases are rejected up front. Vectors inserted after the build return a NonRetryableInsertError rather than being silently lost.
• Metrics: L2 and IP. Cosine runs as IP on vectors that VectorDBBench normalizes (need_normalize_cosine returns True).
• Minimum dimension is 64, a library requirement. Smaller datasets fail at client construction with a clear error.
• Reuse with --skip-drop-old --skip-load: the client compares the manifest against the requested parameters. Only --ef-search may differ from the build. If any build parameter differs, the client refuses the index instead of reporting results for a different index.
• drop_old deletes only the buffer and this index type's files, never the rest of the directory.
• A leftover unbuilt buffer from an interrupted run stops the client with an error instead of loading the data twice.
• Search runs single-threaded per query (num_threads=1), to match the other clients' per-process concurrency.

Options

Option	Default	Notes
--path	required	Holds the buffer, the index and the manifest
--nbits	9	Total bits per coordinate, including the sign bit; 9 (sign + 8) is the fastest kernel
--num-clusters	16	Centroids trained by the library
--m	16	HNSW degree
--ef-construction	200
--ef-search	128	Raised to k when smaller
--seed	100	Rotation and graph seed
--build-threads	0	0 uses every CPU, capped by OMP_NUM_THREADS

Changes

• vectordb_bench/backend/clients/rabitqlib/: new client, configs and CLI command.
• backend/clients/__init__.py, api.py: register DB.RaBitQLib and IndexType.RABITQLIB_HNSW.
• cli/vectordbbench.py: register the rabitqlibhnsw command.
• pyproject.toml: new rabitqlib optional dependency.
• tests/test_rabitqlib.py: 20 tests. They cover:
  • concurrent shuffled inserts;
  • recall on L2 and IP;
  • pickle round trip, and that a spawned process loads the index before init;
  • batch and single search returning the same results;
  • index reuse, and rejection when build parameters change;
  • rejection of inserts after the build, a leftover buffer, the wrong dimension, too small a dimension, and filters;
  • drop_old leaving unrelated files alone.
• README.md: install row and a usage section.

Benchmark: GIST1M (960-d, L2, k=100)

The baseline is Zvec fp32 HNSW, built with the same graph parameters as the RaBitQ index (M=32, ef_construction=200). Both are embedded clients, so they ran under the same process model on the same machine. Machine: 2× Intel Xeon Gold 6418H (96 threads), 1 TB RAM.

RaBitQ: nbits=9, num_clusters=16. RaBitQ built in 106 s; Zvec took about 2 min.

ef_search	Index	Recall@100	QPS c=1	QPS c=8	QPS c=32
128	RaBitQ HNSW	0.8266	2,565	17,894	54,222
128	Zvec HNSW fp32	0.8262	704	3,424	9,259
256	RaBitQ HNSW	0.9124	1,615	11,567	37,085
256	Zvec HNSW fp32	0.9136	395	2,276	5,856
512	RaBitQ HNSW	0.9582	1,021	7,180	22,859
512	Zvec HNSW fp32	0.9615	237	1,292	3,087

	RaBitQ HNSW	Zvec HNSW fp32
Index file	1.38 GB	4.0 GB
Memory per search process	~1.6 GB	~4.3 GB

At equal ef_search the two reach the same recall, as expected for the same graph parameters. At equal recall, RaBitQ gives 3.6–7.4× the QPS with about a third of the memory.

To check the client itself, I queried the saved index directly through rabitqlib and scored it against the GIST ground truth. Recall matched VectorDBBench's to four decimals (0.8266 / 0.9124 / 0.9582).

To reproduce, convert GIST1M to a custom parquet dataset and run:

CASE="--case-type PerformanceCustomDataset --custom-case-name GIST1M \
  --custom-dataset-name GIST --custom-dataset-dir $GIST \
  --custom-dataset-size 1000000 --custom-dataset-dim 960 --custom-dataset-metric-type L2 \
  --custom-dataset-file-count 10 --skip-custom-dataset-use-shuffled --custom-dataset-with-gt \
  --k 100 --num-concurrency 1,8,32"

vectordbbench rabitqlibhnsw --path $IDX/rabitqlib_gist1m --nbits 9 --m 32 \
  --num-clusters 16 --ef-construction 200 --ef-search 256 $CASE
vectordbbench zvec --path $IDX/zvec_gist1m --m 32 --ef-construction 200 --ef-search 256 $CASE

Test plan

• pytest tests/test_rabitqlib.py: 20 passed
• ruff check / ruff format --check on the changed files
• OpenAI 50K end to end (Performance1536D50K): recall 0.9538; QPS 3,385 at c=1 and 11,010 at c=4
• GIST1M end to end against Zvec (table above)
• Recall checked against direct library queries

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: TamimEhsan
To complete the pull request process, please assign xuanyang-cn after the PR has been reviewed.
You can assign the PR to them by writing /assign @xuanyang-cn in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment