Recurrent Residual Quantization (RRQ) for LLMs

[luoyu-intel]

Summary

Recurrent Residual Quantization (RRQ) packs a weight into K sequential INT2 planes (1 base + K−1 residuals). A single checkpoint serves 2 / 4 / 6 / 8-bit weight-only precision — switchable at load/runtime — without re-quantizing, and it is selectable per layer for mixed precision.

Each plane is quantized with the same RTN / SignRound machinery as a standard AutoRound W2A16 model, so per-plane quality parity is guaranteed. The residual planes are stored in the stock AutoRound INT2 layout (qweight / scales / qzeros per plane, indexed *_k), so they reuse existing W2A16 QuantLinear pack/forward kernels with no new runtime.

Status: experimental. Not yet recommended for production deployment.

API

from auto_round import AutoRound, RRQConfig

# --- quantize (base + residual in one pipeline) ---
ar = AutoRound(model, scheme="W2A16", alg_configs=RRQConfig(group_size=128, sym=True))
ar.quantize()
# One call exports BOTH artifacts (residual is written first, then the base is
# packed in place, so the order is handled internally — no manual sequencing):
model, out_dir = rrq.quantize_and_save(output_dir="./out")
# -> ./out/base/      standard W2A16 auto_round model (plane 0), standalone-loadable
# -> ./out/residual/  auto_round:rrq artifact (residual planes 1..K-1)

# --- or build residual from an existing base (Phase 2) ---
from auto_round.export import generate_rrq_residual
generate_rrq_residual(
    "./rrq-base",
    raw_model="Qwen/Qwen3-0.6B",
    output_dir="./rrq-residual",
    group_size=128,
    sym=True,
)

# --- load & switch precision at runtime ---
from auto_round.inference import load_rrq_model
from auto_round.inference.rrq_linear import set_rrq_bits, set_rrq_random_residual

model = load_rrq_model("./rrq-base", "./rrq-residual", active_bits=4, device="xpu")  # uniform
set_rrq_bits(model, 6)                                                                  # switch precision globally

# mixed precision: 50% of layers at 4-bit, rest at 2-bit (~3-bit effective)
m = load_rrq_model(
    "./rrq-base", "./rrq-residual", device="xpu",
    residual_fraction=0.5, residual_seed=0, residual_high_bits=4, residual_low_bits=2,
)
set_rrq_random_residual(m, fraction=0.5, seed=3, high_bits=4, low_bits=2)

Design highlights

• Standard-compatible base. The base plane (plane 0) is an ordinary W2A16 INT2 export that existing runtimes load as-is; RRQ adds residual planes on top.
• Quality parity. Every plane uses the opt-RTN / SignRound pipeline of standard AutoRound. The base plane is IMATrix-weighted opt-RTN (bit-exact with a standard W2A16 base); residual planes seed their scale search with search_optimized_init_scale. With iters>0, per-plane sign-SGD tunes each plane against the block calibration loss while the completed prefix is frozen.
• One artifact, many precisions. effective bits = active_planes × 2 (base=2, +1=4, +2=6, +3=8). set_rrq_bits / set_rrq_random_residual / load_rrq_model(residual_fraction=...) select precision without re-quantizing.
• Naming convention. quant_method stays "auto-round"; RRQ is distinguished by packing_format = "auto_round:rrq" (consistent with how AutoRound differentiates formats like auto_round:auto_gptq).
• Opt-in, not built-in. RRQ is intentionally not in _BUILTIN_ALGORITHM_ORDER. Users opt in via --format auto_round:rrq or alg_configs=RRQConfig(...). disable_opt_rtn=True is enforced in RRQConfig.check_config() (RTN alone only produces one plane).

Changed files

Area	Files
Algorithm	algorithms/quantization/rrq/{__init__,config,quantizer}.py — RRQConfig, RRQRTNQuantizer, RRQSignRoundQuantizer (per-plane sign-SGD); algorithms/registry.py
Export	export/export_to_autoround/export_to_rrq.py (save_quantized_rrq, save_rrq_base_model, generate_rrq_residual); export/formats/backends/rrq.py (auto_round:rrq OutputFormat); export/__init__.py, export/formats/backends/__init__.py
Inference	inference/rrq_linear.py (RRQLinear, set_rrq_bits, set_rrq_random_residual); inference/rrq_model.py (load_rrq_model); inference/backend.py, inference/__init__.py
Integration	__init__.py, autoround.py, compressors/model_free.py, cli/{algorithms,main}.py, utils/common.py
Fail-fast guards	export/export_to_gguf/conversion/base.py, export/export_to_mlx/export.py (reject RRQ residual models, no silent drop)
Bug fix	auto_round_extension/torch/qlinear_torch{,_zp}.py (asym pack self.device → device)
Tests	test/unit/test_cpu/algorithms/test_rrq.py (45 unit tests, incl. lm-eval accuracy)

Validation

• Base-plane parity: the RRQ base plane is bit-exact with a standard W2A16 IMATrix-weighted opt-RTN base at matched calibration.
• Quality parity with W4A16: RRQ reaches or exceeds a standard W4A16 (opt-RTN) model on perplexity / lm-eval at 6/8-bit, and stays within a few points at 4-bit while using a single checkpoint (see the benchmark doc for task-by-task numbers).
• Mixed precision: random per-layer allocation lands within noise of a true 3-bit model while keeping one checkpoint.
• Fail-fast: GGUF / MLX export raise a clear error for RRQ residual models; base/residual config mismatch is rejected at load time.
• Unit tests: pytest test/unit/test_cpu/algorithms/test_rrq.py — 45 passed (config validation, packed-INT2 storage, residual convergence, sym/asym, forward & precision switching, sign-SGD prefix accumulation, export buffer rename, load validation, incremental residual generation, random-residual mixed-precision config, lm-eval accuracy).

Backward compatibility & known limitations

• Default load_rrq_model behavior (uniform active_bits) is unchanged when residual_fraction is not supplied.
• Weight-only (act_bits=16); no activation quantization.
• Fixed 2-bit per plane (no 3/4-bit per-plane planes).
• Reference inference path dequantizes per plane and runs stock W2A16 matmuls — a correctness reference, not a fused kernel (fused packed-INT2 kernels are a follow-up).
• Mixed-precision layer selection is random/uniform; a sensitivity-based selector (IMATrix-weighted output error) is a promising follow-up.

[luoyu-intel]

Qwen3 Quantization Benchmark: W4A16 vs RRQ (Intel Arc Pro B60 XPU)

Test Environment

Item	Description
Model	Qwen/Qwen3-0.6B (596M params, 28 LLM layers, 196/197 quantized) & Qwen/Qwen3-8B (~8B params, 36 LLM layers, 252/253 quantized)
Device	Intel Arc Pro B60 XPU (xpu:1, 24.5 GB)
PyTorch	2.14.0+xpu
Level Zero	1.28.6
auto-round	0.16.0.dev (feat/rrq-phase1, v3 code)
Calibration	NeelNanda/pile-10k, nsamples=512, seqlen=2048, batch_size=8
OPT learning rate	2e-3
Date	2026-09-14

Note: xpu:0 had driver corruption (CPU→XPU DMA degradation); all tests run on xpu:1.
The earlier 8B data (2026-09-12, see "Appendix: Preliminary 8B Results") used suboptimal calibration (nsamples=4, seqlen=64, bs=1) and is provided as reference only.

Test Configurations

Scheme	Name	Description
W4A16 RTN	AutoRound, iters=0	4-bit INT quantization, pure RTN (no sign-SGD optimization)
W4A16 OPT	AutoRound, iters=50, lr=2e-3	4-bit INT quantization with SignRound optimization (50 sign-SGD iterations)
RRQ RTN	RRQ 2+2+2+2, iters=0	4 INT2 planes stacked (2/4/6/8-bit progressive refinement), pure RTN
RRQ OPT	RRQ 2+2+2+2, iters=50, lr=2e-3	4 INT2 planes, each with 50 sign-SGD optimization iterations

Qwen3-0.6B Results (Correct Calibration)

Scheme	Total Time	Tuning Time	Time/Layer	Peak RAM	Peak VRAM
W4A16 RTN (iters=0)	134.61 s (2m 14s)	124.77 s	4.46 s	5.58 GB	4.88 GB
W4A16 OPT (iters=50, lr=2e-3)	187.27 s (3m 07s)	177.57 s	6.34 s	16.84 GB	6.67 GB
RRQ RTN (2+2+2+2, iters=0)	247.36 s (4m 07s)	238.28 s	8.51 s	16.84 GB	6.85 GB
RRQ OPT (2+2+2+2, iters=50, lr=2e-3)	911.04 s (15m 11s)	901.95 s	32.21 s	16.84 GB	6.85 GB

All 4 configurations successfully quantized 196/197 layers (lm_head kept unquantized).

Timing Comparison

RTN vs OPT (within same scheme)

Scheme	RTN Time	OPT Time	OPT/RTN Ratio
W4A16 (1 plane)	134.61 s	187.27 s	1.39×
RRQ 2+2+2+2 (4 planes)	247.36 s	911.04 s	3.68×

W4A16 vs RRQ (within same optimization mode)

Mode	W4A16 Time	RRQ Time	RRQ Overhead
RTN	134.61 s	247.36 s	1.84×
OPT	187.27 s	911.04 s	4.86×

Per-Layer Breakdown

Configuration	Planes	Iters/Plane	Time/Layer	s/Plane/100 Iters
W4A16 RTN	1	0	4.46 s	—
W4A16 OPT	1	50	6.34 s	1.19
RRQ RTN	4	0	8.51 s	—
RRQ OPT	4	50	32.21 s	4.79

• Base RTN cost per layer: ~4.5 s
• Each additional plane adds ~1.35 s to RTN time: (8.51 − 4.46) / 3
• Sign-SGD optimization cost per plane per 100 iters: 1.19 s (W4A16) vs 4.79 s (RRQ)
• RRQ's higher optimization cost is due to residual-plane computation (forward pass through quantized prefix) and the 2-bit quantizer's iterative scale search (181 iterations per group)

Memory Comparison

Scheme	Peak RAM	Peak XPU VRAM
W4A16 RTN	5.58 GB	4.88 GB
W4A16 OPT	16.84 GB	6.67 GB
RRQ RTN	16.84 GB	6.85 GB
RRQ OPT	16.84 GB	6.85 GB

• W4A16 RTN has the lowest memory footprint (5.58 GB RAM) since no calibration data or optimization buffers are needed
• W4A16 OPT, RRQ RTN, and RRQ OPT all use ~16.8 GB RAM — dominated by calibration data (512 samples × 2048 seq × batch 8)
• RRQ OPT slightly exceeds W4A16 OPT in VRAM (6.85 vs 6.67 GB) due to the accumulated residual tensor across 4 planes

Qwen3-8B Results (Preliminary — Suboptimal Calibration)

⚠️ These results use nsamples=4, seqlen=64, batch_size=1 (inadequate calibration).
Re-run with nsamples=512, seqlen=2048, bs=8 pending.

Scheme	Tuning Time	Total Time	Time/Layer	Peak RAM	Peak VRAM
W4A16 RTN (iters=0)	294.48 s (4m 54s)	~5m 00s	8.18 s	22.53 GB	2.13 GB
W4A16 OPT (iters=200)	293.33 s (4m 53s)	304.17 s (5m 04s)	8.15 s	23.60 GB	3.46 GB
RRQ RTN (2+2+2+2, iters=0)	1167.24 s (19m 27s)	1173.84 s (19m 33s)	32.24 s	13.65 GB	3.66 GB
RRQ OPT (2+2+2+2, iters=200)	7094.89 s (1h 58m 14s)	7100.74 s (1h 58m 20s)	196.77 s	14.30 GB	5.01 GB

All 4 configurations successfully quantized 252/253 layers (lm_head kept unquantized). Device: xpu:1.

8B Timing Comparison (Preliminary)

RTN vs OPT (within same scheme)

Scheme	RTN Time	OPT Time	OPT/RTN Ratio
W4A16 (1 plane)	294.48 s	293.33 s	1.00× (no measurable difference)
RRQ 2+2+2+2 (4 planes)	1167.24 s	7094.89 s	6.07×

W4A16 OPT on 8B shows no measurable time increase over RTN (with only 4 samples) — the sign-SGD iterations are negligible relative to model loading and inference overhead. With proper calibration (nsamples=512), the OPT overhead should become visible.

W4A16 vs RRQ (within same optimization mode)

Mode	W4A16 Time	RRQ Time	RRQ Overhead
RTN	294.48 s	1167.24 s	3.96×
OPT	293.33 s	7094.89 s	24.19×

8B Per-Layer Breakdown (Preliminary)

Configuration	Planes	Iters/Plane	Time/Layer	s/Plane/100 Iters
W4A16 RTN	1	0	8.18 s	—
W4A16 OPT	1	200	8.15 s	~0 (no measurable diff)
RRQ RTN	4	0	32.24 s	—
RRQ OPT	4	200	196.77 s	~12.2

• Base RTN cost per layer: ~8.2 s (8B) vs ~4.5 s (0.6B) — scales ~1.8× with tensor size
• RRQ RTN overhead per additional plane: (32.24 − 8.18) / 3 = 8.02 s (8B) vs 1.35 s (0.6B) — 5.9× larger
• Sign-SGD optimization cost per plane per 100 iters: ~12.2 s (8B RRQ, 200 iters) vs 4.79 s (0.6B RRQ, 50 iters) — note different iteration counts

8B Memory Comparison (Preliminary)

Scheme	Peak RAM	Peak VRAM	Notes
W4A16 RTN	22.53 GB	2.13 GB	Model weights in RAM, minimal on XPU
W4A16 OPT	23.60 GB	3.46 GB	+ gradient/optimizer buffers for 1 plane
RRQ RTN	13.65 GB	3.66 GB	4 planes quantized sequentially, lower RAM
RRQ OPT	14.30 GB	5.01 GB	+ gradient buffers for 4 small planes

• RRQ RTN uses 40% less RAM than W4A16 RTN (13.65 vs 22.53 GB) — 2-bit weights are smaller
• W4A16 OPT uses 67% more RAM than RRQ RTN (23.60 vs 13.65 GB)
• RRQ OPT uses 40% less RAM than W4A16 OPT (14.30 vs 23.60 GB)
• RRQ OPT uses 45% more VRAM than W4A16 OPT (5.01 vs 3.46 GB)

Summary

Qwen3-0.6B (Correct Calibration: nsamples=512, seqlen=2048, bs=8)

Metric	W4A16 RTN	W4A16 OPT	RRQ RTN	RRQ OPT
Total time	2m 14s	3m 07s	4m 07s	15m 11s
Effective bits	4	4	8 (2+2+2+2)	8 (2+2+2+2)
Bits per plane	4	4	2	2
Number of planes	1	1	4	4
Sign-SGD iters	0	50	0	50×4
Peak RAM	5.58 GB	16.84 GB	16.84 GB	16.84 GB
Peak VRAM	4.88 GB	6.67 GB	6.85 GB	6.85 GB

Qwen3-8B (Preliminary: nsamples=4, seqlen=64, bs=1)

Metric	W4A16 RTN	W4A16 OPT	RRQ RTN	RRQ OPT
Total time	~5m 00s	5m 04s	19m 33s	1h 58m 20s
Tuning time	294.48 s	293.33 s	1167.24 s	7094.89 s
Time/layer	8.18 s	8.15 s	32.24 s	196.77 s
Effective bits	4	4	8 (2+2+2+2)	8 (2+2+2+2)
Bits per plane	4	4	2	2
Number of planes	1	1	4	4
Sign-SGD iters	0	200	0	200×4
Peak RAM	22.53 GB	23.60 GB	13.65 GB	14.30 GB
Peak VRAM	2.13 GB	3.46 GB	3.66 GB	5.01 GB

Conclusions

1. RRQ code is correct: All configurations completed successfully on XPU (0.6B: 4/4, 8B: 4/4)

2. RRQ RTN overhead is ~1.8–4× W4A16 RTN depending on model size:

   • 0.6B: 1.84× (247s vs 135s)
   • 8B (preliminary): 3.96× (1167s vs 294s)
   • Overhead grows with model size because each residual plane must forward through the full accumulated (larger) tensor

3. RRQ OPT overhead is ~4.9–24× W4A16 OPT depending on model size and calibration:

   • 0.6B (512 samples, 50 iters): 4.86×
   • 8B (4 samples, 200 iters, preliminary): 24.19×
   • The 24× on 8B is inflated by the 200-iter setting (vs 50-iter for 0.6B) and minimal calibration data
   • Primary cost drivers: (a) 4× iteration count, (b) 2-bit quantizer's 181-iteration scale search per group, (c) larger accumulated forward pass

4. Sign-SGD cost per iteration (0.6B, 50 iters):

   • W4A16: (6.34 − 4.46) / 1 plane = 1.88 s/plane for 50 iters
   • RRQ: (32.21 − 8.51) / 4 planes = 5.93 s/plane for 50 iters
   • RRQ per-iteration cost: 3.2× higher (due to 2-bit scale search + residual forward)

5. Memory profile is similar across all OPT/RTN-RRQ configs (0.6B): all dominated by calibration data (~16.8 GB RAM). The model itself is small enough that optimization buffer differences are minor.

6. torch.compile is NOT beneficial on XPU (tested separately):

   • quant_tensor: 0.99× (slightly slower)
   • F.linear fwd: 2.8× slower
   • fwd+backward: 2× slower
   • Root cause: IPEX/Level Zero Inductor backend not optimized for these ops

7. Practical recommendation: For 8B+ models on XPU, use RRQ RTN (iters=0) for speed, or reduce iters to 30–50 for RRQ OPT. The full 200-iter OPT on 8B would take ~4–8 hours with proper calibration.

Appendix: Preliminary 8B Results (nsamples=4, seqlen=64, bs=1)

These were the first 8B runs (2026-09-12/13) using suboptimal calibration parameters. They are kept for reference but should be superseded by a proper run with nsamples=512, seqlen=2048, bs=8.

Key observations from preliminary data:

• W4A16 OPT did NOT show measurable overhead over RTN (293s vs 294s) — because with only 4 calibration samples, the optimization loop has almost no work to do
• RRQ OPT at 200 iters took 1h58m — dominated by 800 forward+backward passes with 181-iter scale search
• With proper calibration (512 samples), W4A16 OPT will show meaningful overhead, and the RRQ/W4 ratio will be more representative

[luoyu-intel]

implementation of #2300

[AutoRoundBot]

/azp run Unit-Test-CUDA-AutoRound

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Copilot]

🟡 Changes recommended

load_rrq_model can silently leave some packed base layers as uninitialized/random nn.Linear weights when residual planes are missing or skipped, which is correctness-critical for inference.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds Recurrent Residual Quantization (RRQ) to AutoRound, enabling a single INT2-base checkpoint plus packed INT2 residual planes to support dynamic 2/4/6/8-bit (and mixed-precision) inference via a new auto_round:rrq residual artifact and corresponding loader/runtime modules.

Changes:

• Introduces RRQ algorithm config + quantizers (RTN and per-plane SignRound tuning) and registers it in the algorithm registry.
• Adds RRQ residual export format (auto_round:rrq) and inference-time composition (load_rrq_model, RRQLinear, precision switching utilities).
• Updates quantized linear kernels to expose _dequantize() for reuse, plus adds extensive CPU unit tests and user scripts; updates README(+CN) and ignores rrq_output/.

File summaries

File	Description
test/unit/test_cpu/algorithms/test_rrq.py	Comprehensive CPU unit tests for RRQ config, packing, reconstruction, inference switching, and Phase 2/3 behaviors.
test_rrq_qwen3_06b.py	Standalone script to quantize Qwen3-0.6B with RRQ and verify base/residual layout + (optional) load/forward.
test_rrq_lm_eval.py	Standalone script to run lm-eval across RRQ bit-widths (base+residual).
README.md	Adds RRQ announcement to “What’s New”.
README_CN.md	Chinese counterpart update for the RRQ “What’s New” entry.
auto_round/utils/common.py	Adds auto_round:rrq to supported formats list.
auto_round/inference/rrq_model.py	New loader that merges base + residual artifacts into an RRQ-enabled model by replacing layers with RRQLinear.
auto_round/inference/rrq_linear.py	New RRQLinear module and helpers to set uniform or random mixed precision across layers.
auto_round/inference/backend.py	Adds RRQ format constant (RRQ_FORMAT).
auto_round/export/formats/backends/rrq.py	New OutputFormat backend for auto_round:rrq residual export.
auto_round/export/formats/backends/__init__.py	Exposes RRQFormat in backend imports/exports.
auto_round/export/export_to_mlx/export.py	Fail-fast guard rejecting RRQ residual models for MLX export.
auto_round/export/export_to_gguf/conversion/base.py	Fail-fast guard rejecting RRQ residual models for GGUF export.
auto_round/export/export_to_autoround/export_to_rrq.py	Implements RRQ residual serialization + Phase 2 residual generation from base+raw weights.
auto_round/compressors/model_free.py	Frees packed shard tensors earlier to improve memory reclamation; tweaks a log message.
auto_round/cli/algorithms.py	Improves CLI arg merge logic to avoid mismatching boolean optional arguments with shared dest.
auto_round/autoround.py	Forces RRQ to route through calibrated path (disables model-free path) to avoid dropping residual planes.
auto_round/algorithms/registry.py	Registers RRQ config/quantizer modules and adds rrq to built-in algorithm order.
auto_round/algorithms/quantization/rrq/quantizer.py	Core RRQ quantizers (RTN multi-plane + SignRound per-plane tuning with frozen prefix) and packing logic.
auto_round/algorithms/quantization/rrq/config.py	RRQConfig implementation (fixed INT2 planes, tuning params, calibration requirement).
auto_round/algorithms/quantization/rrq/__init__.py	RRQ module exports.
auto_round/__init__.py	Exposes RRQConfig and lazily exports load_rrq_model / generate_rrq_residual.
auto_round_extension/torch/qlinear_torch.py	Fixes device usage in packing; adds _dequantize() helper and adjusts g_idx logic.
auto_round_extension/torch/qlinear_torch_zp.py	Adds _dequantize() helper and adjusts g_idx logic (symmetric/GPTQ-style).
.gitignore	Ignores rrq_output/ directory.

Review details

• Files reviewed: 24/25 changed files
• Comments generated: 7
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[a32543254]

LGTM

[AutoRoundBot]

CI Failure Analysis Report (Unit-Test-AutoRound)

• Commit: 6cd1c74
• Run: https://dev.azure.com/lpot-inc/neural-compressor/_build/results?buildId=77599
• Totals: 1 failed test(s), 1 cluster(s) (0 known / 1 new)

New Issues (1)

No.1 — 1 occurrence(s)

🔍 AssertionError: 'model.visual.merger.linear_fc2' should be in modules_to_not_convert. Got: []

📝 Basic info

• Affected tests (1): TestAutoRound.test_autoawq_qwen3_vl_infer
• Logs (1): unittest_test_export.log

🖥️ Log excerpt

unit/test_cpu/export/test_export.py:618: in test_autoawq_qwen3_vl_infer
    assert (
E   AssertionError: 'model.visual.merger.linear_fc2' should be in modules_to_not_convert. Got: []
E   assert 'model.visual.merger.linear_fc2' in []

✨ AI analysis

• Category: Unknown
• Confidence: low

Notes

• Only top-3 issues receive AI analysis.
• To request an additional fix from Copilot, use "Quote reply" on the PR comment and @mention Copilot.

[luoyu-intel]

Qwen3 Quantization Benchmark Report (0.6B & 8B)

Test Environment

Item	Description
Model	Qwen/Qwen3-0.6B (596M params, ~2.22 GB float32) & Qwen/Qwen3-8B (~8B params, 36 layers)
Device	Intel Arc Pro B60 XPU (xpu:1, 24.5 GB)
PyTorch	2.14.0+xpu
Level Zero	1.28.6
auto-round	0.16.0.dev (feat/rrq-phase1)
Calibration	NeelNanda/pile-10k, 4 samples, seqlen=64, batch_size=1
Date	2026-09-12 – 2026-09-13

Note: xpu:0 driver state was corrupted (CPU→XPU DMA degradation); all tests completed on xpu:1.

Test Configurations

Scheme	Name	Description
W4A16 RTN	AutoRound, iters=0	4-bit INT quantization, pure RTN (no sign-SGD optimization)
W4A16 OPT	AutoRound, iters=200 (default)	4-bit INT quantization with SignRound optimization (200 sign-SGD iterations)
RRQ RTN	RRQ 2+2+2+2, iters=0	4 INT2 planes stacked (2/4/6/8-bit progressive refinement), pure RTN
RRQ OPT	RRQ 2+2+2+2, iters=200 (default)	4 INT2 planes, each with 200 sign-SGD optimization iterations

Key Results

Scheme	Total Time	Tuning Time	Time/Layer	Peak RAM	Peak VRAM
W4A16 RTN (iters=0)	65.45 s (1m 05s)	59.41 s	2.12 s	5.57 GB	0.11 GB
W4A16 OPT (iters=200)	90.81 s (1m 31s)	83.60 s	2.99 s	15.58 GB	0.30 GB
RRQ RTN (2+2+2+2, iters=0)	180.16 s (3m 00s)	174.22 s	6.16 s	6.05 GB	0.21 GB
RRQ OPT (2+2+2+2, iters=200)	876.42 s (14m 36s)	870.91 s	31.01 s	6.91 GB	0.43 GB

All 4 configurations successfully quantized 196/197 layers (lm_head kept unquantized).

Timing Comparison

RTN vs OPT (within same scheme)

Scheme	RTN Time	OPT Time	Speedup (RTN/OPT)	OPT/RTN Ratio
W4A16 (1 plane)	65.45 s	90.81 s	0.72×	1.39×
RRQ 2+2+2+2 (4 planes)	180.16 s	876.42 s	0.21×	4.86×

W4A16 vs RRQ (within same optimization mode)

Mode	W4A16 Time	RRQ Time	RRQ Overhead
RTN	65.45 s	180.16 s	2.75×
OPT	90.81 s	876.42 s	9.65×

Per-Layer Breakdown

Configuration	Planes	Iters/Plane	Time/Layer	s/Plane/100 Iters
W4A16 RTN	1	0	2.12 s	—
W4A16 OPT	1	200	2.99 s	0.44
RRQ RTN	4	0	6.16 s	—
RRQ OPT	4	200	31.01 s	6.92

• Base RTN cost per layer: ~2.1 s
• Each additional plane adds ~1.3 s to RTN time: (6.16 − 2.12) / 3
• Sign-SGD optimization cost per plane per 100 iters: 0.44 s (W4A16) vs 6.92 s (RRQ)
• RRQ's higher optimization cost is due to residual-plane computation (forward pass through quantized prefix)

Memory Comparison

Scheme	Peak RAM	Peak XPU VRAM
W4A16 RTN	5.57 GB	0.11 GB
W4A16 OPT	15.58 GB	0.30 GB
RRQ RTN	6.05 GB	0.21 GB
RRQ OPT	6.91 GB	0.43 GB

• W4A16 OPT peak RAM (15.58 GB) is 56% higher than RRQ OPT (6.91 GB) — the single 4-bit plane requires larger gradient buffers
• RRQ's four 2-bit planes have lower per-plane optimization overhead, resulting in lower total memory despite 4× the number of planes

Qwen3-8B Results

Scheme	Tuning Time	Total Time	Time/Layer	Peak RAM	Peak VRAM
W4A16 RTN (iters=0)	294.48 s (4m 54s)	~5m 00s	8.18 s	22.53 GB	2.13 GB
W4A16 OPT (iters=200)	293.33 s (4m 53s)	304.17 s (5m 04s)	8.15 s	23.60 GB	3.46 GB
RRQ RTN (2+2+2+2, iters=0)	1167.24 s (19m 27s)	1173.84 s (19m 33s)	32.24 s	13.65 GB	3.66 GB
RRQ OPT (2+2+2+2, iters=200)	7094.89 s (1h 58m 14s)	7100.74 s (1h 58m 20s)	196.77 s	14.30 GB	5.01 GB

All 4 configurations successfully quantized 252/253 layers (lm_head kept unquantized). Device: xpu:1.

8B Timing Comparison

RTN vs OPT (within same scheme)

Scheme	RTN Time	OPT Time	OPT/RTN Ratio
W4A16 (1 plane)	294.48 s	293.33 s	1.00× (no measurable difference)
RRQ 2+2+2+2 (4 planes)	1167.24 s	7094.89 s	6.07×

W4A16 OPT on 8B shows no measurable time increase over RTN — likely because the single 4-bit plane's sign-SGD iterations are fast relative to the large tensor size. In contrast, RRQ's 4 residual planes each require 200 iterations through the accumulated (larger) tensor, amplifying the cost 6×.

W4A16 vs RRQ (within same optimization mode)

Mode	W4A16 Time	RRQ Time	RRQ Overhead
RTN	294.48 s	1167.24 s	3.96×
OPT	293.33 s	7094.89 s	24.19×

Cross-Model RRQ/W4 Ratio

Mode	Qwen3-0.6B (28 layers)	Qwen3-8B (36 layers)
RTN	2.75×	3.96×
OPT	9.65×	24.19×

The RRQ OPT overhead grows dramatically with model size (9.65× → 24.19×). This is because each residual-plane forward pass must process the full accumulated tensor (which grows with model size), and 4 planes × 200 iterations compound this effect super-linearly.

8B Per-Layer Breakdown

Configuration	Planes	Iters/Plane	Time/Layer	s/Plane/100 Iters
W4A16 RTN	1	0	8.18 s	—
W4A16 OPT	1	200	8.15 s	~0 (no measurable diff)
RRQ RTN	4	0	32.24 s	—
RRQ OPT	4	200	196.77 s	~44.7

• Base RTN cost per layer: ~8.2 s (8B) vs ~2.1 s (0.6B) — scales with tensor size
• RRQ RTN overhead per additional plane: (32.24 − 8.18) / 3 = 8.02 s (8B) vs 1.35 s (0.6B)
• Sign-SGD optimization cost per plane per 100 iters: ~44.7 s (8B RRQ) vs 6.92 s (0.6B RRQ) — 6.5× larger

8B Memory Comparison

Scheme	Peak RAM	Peak VRAM	Notes
W4A16 RTN	22.53 GB	2.13 GB	Model + calibration on XPU
W4A16 OPT	23.60 GB	3.46 GB	Gradient buffers for 1 large plane
RRQ RTN	13.65 GB	3.66 GB	No gradient buffers, 4 planes in flight
RRQ OPT	14.30 GB	5.01 GB	Gradient buffers for 4 small planes

• W4A16 OPT uses 65% more RAM than RRQ RTN (23.6 vs 13.65 GB) but 45% less VRAM (3.46 vs 5.01 GB)
• RRQ OPT uses 39% less RAM than W4A16 OPT (14.3 vs 23.6 GB) but 45% more VRAM (5.01 vs 3.46 GB)
• The tradeoff: RRQ's four 2-bit planes keep individual gradient buffers small, but the accumulated residual tensor in VRAM grows larger during optimization

Summary (Qwen3-0.6B)

Metric	W4A16 RTN	W4A16 OPT	RRQ RTN	RRQ OPT
Total time	1m 05s	1m 31s	3m 00s	14m 36s
Effective bits	4	4	8 (2+2+2+2)	8 (2+2+2+2)
Bits per plane	4	4	2	2
Number of planes	1	1	4	4
Sign-SGD iters	0	200	0	200×4
Peak RAM	5.57 GB	15.58 GB	6.05 GB	6.91 GB
Peak VRAM	0.11 GB	0.30 GB	0.21 GB	0.43 GB

Summary (Qwen3-8B)

Metric	W4A16 RTN	W4A16 OPT	RRQ RTN	RRQ OPT
Total time	~5m 00s	5m 04s	19m 33s	1h 58m 20s
Tuning time	294.48 s	293.33 s	1167.24 s	7094.89 s
Time/layer	8.18 s	8.15 s	32.24 s	196.77 s
Effective bits	4	4	8 (2+2+2+2)	8 (2+2+2+2)
Bits per plane	4	4	2	2
Number of planes	1	1	4	4
Sign-SGD iters	0	200	0	200×4
Peak RAM	22.53 GB	23.60 GB	13.65 GB	14.30 GB
Peak VRAM	2.13 GB	3.46 GB	3.66 GB	5.01 GB

[AutoRoundBot]

/azp run Unit-Test-CUDA-AutoRound

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Copilot]

🟡 Changes recommended

Critical API, export sequencing, metadata, quantization, and inference issues remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (12)

auto_round/algorithms/quantization/rrq/quantizer.py:446

• This residual packing path assumes original_weight is [out_features, in_features], but a Transformers Conv1D weight is stored transposed. It constructs the residual QuantLinear using the raw dimensions, while the standard exporter transposes Conv1D before packing, so the residual qweight shape/semantics will not match the base artifact and can fail in load_rrq_model. Apply the same Conv1D normalization as the standard exporter before calling _pack_plane.

                # Residual plane: store packed INT2 (W2A16 layout) so the
                # on-disk artifact is a standard single-plane INT2 layout.
                in_features = original_weight.shape[1]
                qweight, scales, qzeros = self._pack_plane(quantized, scale, zp, bits, group_size, in_features)

auto_round/algorithms/quantization/rrq/quantizer.py:626

• The configured dynamic_max_gap early-stop behavior is not implemented here: after tracking the best loss, this branch executes pass for every positive gap. RRQ tuning therefore always runs all iterations even when users request the same stopping criterion supported by SignRound; add the best-iteration tracking and break condition before merging this path.

            if not self.not_use_best_mse and 0 < self.dynamic_max_gap:
                # Keep the same early-stop contract as SignRound; the default
                # -1 disables this path.
                pass

auto_round/export/export_to_autoround/export_to_rrq.py:260

• save_quantized_rrq() always pops and renames the module buffers in place, but RRQFormat.save_quantized() accepts the standard inplace argument and does not pass it through. Therefore save_quantized(..., inplace=False) still mutates the quantized model and can make a subsequent base export lose the residual buffers, violating the exporter contract.

    # Build a state dict containing *only* the packed residual planes, and
    # rename the in-memory ``rrq_*_k`` buffers to the on-disk ``*_k`` names.
    residual_state: dict[str, torch.Tensor] = {}
    for name, module in model.named_modules():
        if not hasattr(module, "rrq_total_planes"):
            continue
        for k in range(1, module.rrq_total_planes):
            for src, dst in (
                (f"rrq_qweight_{k}", f"qweight_{k}"),
                (f"rrq_scales_{k}", f"scales_{k}"),
                (f"rrq_qzeros_{k}", f"qzeros_{k}"),
            ):
                if src in module._buffers:
                    module._buffers[dst] = module._buffers.pop(src)

auto_round/export/export_to_autoround/export_to_rrq.py:561

• The incremental generator has the same Conv1D orientation mismatch: raw GPT-2-style weights are [in, out], whereas QuantLinear.forward(identity).T here is [out, in]. The subtraction in the next step will either fail for non-square layers or generate transposed residuals, so this path cannot generate valid residuals for Conv1D-based LLMs.

        # Dequant by running forward on identity
        identity = torch.eye(in_features, dtype=torch.float32)
        with torch.no_grad():
            out = ql.forward(identity)  # (in_features, out_features)
        W_dequant_base = out.T.to(torch.float32)  # (out_features, in_features)

auto_round/export/export_to_autoround/export_to_rrq.py:411

• The public device argument is documented as the computation device, but this implementation forces dequantization, identity inputs, quantization, and packing to CPU (ql.to("cpu"), device="cpu"). Passing a GPU/XPU device therefore has no effect and makes residual generation for a large model unnecessarily CPU-bound. Thread the requested device through the per-layer quantization and packing path.

    device: Union[str, torch.device] = "cpu",

auto_round/export/export_to_autoround/export_to_rrq.py:266

• If the standard base exporter has already replaced the RRQ layers, this loop finds no rrq_* buffers, yet the code still writes an artifact. The resulting residual directory looks valid but load_rrq_model later fails because every base layer is missing residual planes. Reject an empty residual state before serializing it.

    # Serialize the residual state dict (safetensors by default, torch otherwise).
    _save_state_dict_sharded(residual_state, output_dir, safe_serialization)

auto_round/export/formats/backends/rrq.py:72

• inplace is part of this backend's save contract, but it is never forwarded to save_quantized_rrq; that helper renames buffers directly on the original model. Consequently save_quantized(..., inplace=False) still mutates and removes the rrq_* buffers, which can break later exports or use of the source model. Thread the flag through and clone before renaming, or reject unsupported inplace=False explicitly.

            inplace: Whether to modify the model in place.

auto_round/inference/rrq_model.py:304

• Catching every Exception here turns real load failures (for example missing/corrupt weights, an invalid config, or OOM) into a config-only model with randomly initialized non-quantized parameters. load_rrq_model can therefore return a plausible-looking but incorrect model. Restrict the fallback to a known packed-loading incompatibility and otherwise re-raise, or load non-quantized weights through a controlled path.

    except Exception:  # pragma: no cover - fall back to architecture-only load
        logger.warning(
            "from_pretrained(%s) failed; loading architecture without weights.",

auto_round/inference/rrq_model.py:324

• Skipping a missing module leaves that packed base layer unreconstructed, but the function only fails when all layers are skipped. The returned model can therefore contain a normal/random nn.Linear for one or more checkpoint layers while silently omitting their residuals. Treat any missing eligible module as an incompatible checkpoint and raise instead of continuing.

        try:
            base_model.get_submodule(layer_name)
        except AttributeError:
            logger.warning(f"Base module {layer_name!r} not found; skipping.")
            continue

test/unit/test_cpu/algorithms/test_rrq.py:1020

• set_rrq_bits only traverses RRQLinear modules, but AutoRound.quantize() returns the original quantized linear modules with rrq_* buffers; the conversion to RRQLinear happens in load_rrq_model. This call therefore only logs a warning and the following supposed 4-bit evaluation still uses the 8-bit/full-plane model, so the regression test does not exercise precision switching.

        set_rrq_bits(model, 4)

test/unit/test_cpu/algorithms/test_rrq.py:200

• This test does not exercise the new AutoRound.__new__ format auto-selection at all; it only resolves the registry entry directly. A regression in the constructor/format path would still pass. Exercise the constructor or CLI path and assert both the selected RRQ config and resolved RRQ format.

        # Simulate what _CompressorBuilder.__new__ does:
        # when alg_configs is None and format is auto_round:rrq, it picks "rrq".
        from auto_round.algorithms.registry import resolve_alg_config, resolve_algorithm_names

        config = resolve_alg_config("rrq")

test/unit/test_cpu/algorithms/test_rrq.py:198

• resolve_algorithm_names is imported but never used in this test; the unused local import will be flagged by the repository's Ruff/Pyflakes checks. Remove it (or use it in the assertion).

        from auto_round.algorithms.registry import resolve_alg_config, resolve_algorithm_names

• Files reviewed: 26/27 changed files
• Comments generated: 11
• Review effort level: Lite

[wenhuach21]

Could you upload a model to the Intel space and add the link here as an example model? Please also include detailed generation commands and an inference script in the model card. I think this would make things clearer for users.