Skip to content

pipelines: forward --quantization to the multi-GPU runners - #280

Open
Boning (wwelder) wants to merge 1 commit into
Lightricks:mainfrom
wwelder:fix/mgpu-forward-quantization
Open

pipelines: forward --quantization to the multi-GPU runners#280
Boning (wwelder) wants to merge 1 commit into
Lightricks:mainfrom
wwelder:fix/mgpu-forward-quantization

Conversation

@wwelder

Copy link
Copy Markdown

Problem

The three multi-GPU CLIs — distilled_mgpu.py, ti2vid_two_stages_mgpu.py, ti2vid_two_stages_hq_mgpu.py — never pass quantization to controller.start(). setup() therefore always takes its fallback:

quantization_policy = (
    quantization() if quantization is not None else _build_fp8_cast_policy(model_paths.transformer())
)

so --quantization is silently ignored on every multi-GPU path. The single-GPU pipelines do forward it (quantization=args.quantization in distilled.py), so the flag works there.

With an NVFP4 checkpoint the run then fails inside the fp8 loader, which expects a scalar scale and rejects NVFP4's per-block scales:

SymmetricRunnerError: Unsupported scale shape (4096, 256) for transformer_blocks.0.attn1.to_k.weight

Reproduce with the documented nvfp4-prequant invocation, but on the mgpu entry point:

python -m ltx_pipelines.distilled_mgpu \
    --transformer-path .../ltx-2.5-22b-distilled-transformer-nvfp4.safetensors \
    --text-encoder-path .../gemma4-12b-with-proj-ltx-2.5-bf16.safetensors \
    --video-vae-path .../ltx-2.5-video-vae-conv-bf16.safetensors \
    --audio-vae-path .../ltx-2.5-audio-vae-bf16.safetensors \
    --spatial-upsampler-path .../ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0.safetensors \
    --quantization nvfp4-prequant --prompt "..." --output-path out.mp4

Why args.quantization can't just be forwarded

setup() documents its parameter as "a picklable zero-arg builder (built per worker, post-spawn)" and types it Callable[[], QuantizationPolicy] | None, whereas args.quantization is an already-built QuantizationPolicy. The policy is constructed in the parent and would have to survive spawn; the builder deliberately runs inside each worker instead.

_resolve_quantization already has both kind and the resolved checkpoint path, but discards them after building the policy. This adds namespace.quantization_builder, a functools.partial over QuantizationKind.to_policy. Enum members and bound methods pickle by reference, so the partial round-trips through spawn cleanly.

Behaviour

  • Single-GPU pipelines: unchanged, they keep reading args.quantization.
  • No --quantization: quantization_builder is None, and the runners keep their existing fp8-cast default.
  • _resolve_quantization runs unconditionally in the parser's parse_args override, alongside _resolve_model_paths, so the attribute always exists wherever args.model_paths already does.

Verification

Tested on 2× and 4× RTX PRO 6000 Blackwell (sm_120) with ltx-2.5-22b-distilled-transformer-nvfp4.safetensors plus the bf16 text encoder and conv video VAE. Before: every run failed with the scale-shape error above. After: the NVFP4 checkpoint loads and generates normally — dozens of 15 s clips at 576×832 and 704×1280, across sequence-parallel world sizes 2 and 4.

The three `*_mgpu.py` CLIs never passed `quantization` to
`controller.start()`, so `setup()` always took its `_build_fp8_cast_policy`
fallback and `--quantization` was silently ignored on every multi-GPU path.
With an NVFP4 checkpoint the run then dies in the fp8 loader, which expects a
scalar scale and rejects NVFP4's per-block scales:

    SymmetricRunnerError: Unsupported scale shape (4096, 256)
    for transformer_blocks.0.attn1.to_k.weight

`args.quantization` cannot be forwarded directly: `setup()` wants a picklable
zero-arg builder, invoked inside each spawned worker, while `args.quantization`
is an already-built policy. `_resolve_quantization` now also exposes
`quantization_builder`, a `functools.partial` over `QuantizationKind.to_policy`
that pickles cleanly, and the multi-GPU CLIs forward that.

Single-GPU pipelines are unaffected: they keep reading `args.quantization`.
When `--quantization` is absent, `quantization_builder` is None and the runners
keep their existing fp8-cast default.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant