pipelines: forward --quantization to the multi-GPU runners - #280
Open
Boning (wwelder) wants to merge 1 commit into
Open
pipelines: forward --quantization to the multi-GPU runners#280Boning (wwelder) wants to merge 1 commit into
Boning (wwelder) wants to merge 1 commit into
Conversation
The three `*_mgpu.py` CLIs never passed `quantization` to
`controller.start()`, so `setup()` always took its `_build_fp8_cast_policy`
fallback and `--quantization` was silently ignored on every multi-GPU path.
With an NVFP4 checkpoint the run then dies in the fp8 loader, which expects a
scalar scale and rejects NVFP4's per-block scales:
SymmetricRunnerError: Unsupported scale shape (4096, 256)
for transformer_blocks.0.attn1.to_k.weight
`args.quantization` cannot be forwarded directly: `setup()` wants a picklable
zero-arg builder, invoked inside each spawned worker, while `args.quantization`
is an already-built policy. `_resolve_quantization` now also exposes
`quantization_builder`, a `functools.partial` over `QuantizationKind.to_policy`
that pickles cleanly, and the multi-GPU CLIs forward that.
Single-GPU pipelines are unaffected: they keep reading `args.quantization`.
When `--quantization` is absent, `quantization_builder` is None and the runners
keep their existing fp8-cast default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The three multi-GPU CLIs —
distilled_mgpu.py,ti2vid_two_stages_mgpu.py,ti2vid_two_stages_hq_mgpu.py— never passquantizationtocontroller.start().setup()therefore always takes its fallback:so
--quantizationis silently ignored on every multi-GPU path. The single-GPU pipelines do forward it (quantization=args.quantizationindistilled.py), so the flag works there.With an NVFP4 checkpoint the run then fails inside the fp8 loader, which expects a scalar scale and rejects NVFP4's per-block scales:
Reproduce with the documented
nvfp4-prequantinvocation, but on the mgpu entry point:python -m ltx_pipelines.distilled_mgpu \ --transformer-path .../ltx-2.5-22b-distilled-transformer-nvfp4.safetensors \ --text-encoder-path .../gemma4-12b-with-proj-ltx-2.5-bf16.safetensors \ --video-vae-path .../ltx-2.5-video-vae-conv-bf16.safetensors \ --audio-vae-path .../ltx-2.5-audio-vae-bf16.safetensors \ --spatial-upsampler-path .../ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0.safetensors \ --quantization nvfp4-prequant --prompt "..." --output-path out.mp4Why
args.quantizationcan't just be forwardedsetup()documents its parameter as "a picklable zero-arg builder (built per worker, post-spawn)" and types itCallable[[], QuantizationPolicy] | None, whereasargs.quantizationis an already-builtQuantizationPolicy. The policy is constructed in the parent and would have to survivespawn; the builder deliberately runs inside each worker instead._resolve_quantizationalready has bothkindand the resolved checkpoint path, but discards them after building the policy. This addsnamespace.quantization_builder, afunctools.partialoverQuantizationKind.to_policy. Enum members and bound methods pickle by reference, so the partial round-trips throughspawncleanly.Behaviour
args.quantization.--quantization:quantization_builderisNone, and the runners keep their existing fp8-cast default._resolve_quantizationruns unconditionally in the parser'sparse_argsoverride, alongside_resolve_model_paths, so the attribute always exists whereverargs.model_pathsalready does.Verification
Tested on 2× and 4× RTX PRO 6000 Blackwell (sm_120) with
ltx-2.5-22b-distilled-transformer-nvfp4.safetensorsplus the bf16 text encoder and conv video VAE. Before: every run failed with the scale-shape error above. After: the NVFP4 checkpoint loads and generates normally — dozens of 15 s clips at 576×832 and 704×1280, across sequence-parallel world sizes 2 and 4.