docs(moe): re-validate CUDA fused-MoE Dff cap on MLX 0.32.1 - #721
Merged
Conversation
Issue #330 / PR #711 left a pending re-validation note in docs/benchmark_results/fused-moe-decode-kernel-design.md because the GB10 sweep behind the CUDA FUSED_MOE_MAX_DFF_CUDA default (8192) was measured on MLX pin e9463bb, and the pin has since advanced to 0.32.1 (57c66cac, #703/#704). Re-ran the same harness on GB10 (sm_121, CUDA 13.0) under the current 0.32.1 pin: mlxcel-bench-decode, prompt "Hello, how are you today?", 100 decode tokens after a 20-token warmup, median of 3 runs, MLXCEL_FUSED_MOE_MAX_DFF=1 (gather_qmm fallback) vs =20000 (fused) across lfm2-8b-a1b (Dff 1792), phi-3.5-moe (6400), llama-4-scout-17b (8192), mixtral-8x7b (14336). The ratio=1.0 crossover interpolates to ~7672, close to the prior ~8000 given the run-to-run noise the 2026-07-03 sweep already flagged at the same Dff point. FUSED_MOE_MAX_DFF_CUDA in src/models/switch_layers.rs stays at 8192; this is a documentation-only change. Adds a 2026-07-10 addendum with the new sweep table and conclusion, amends the 2026-07-09 addendum's pending note to point at it while keeping it as historical provenance, updates the env-var table row, and commits the raw per-run data as benchmarks/cuda_gb10_issue712_fused_moe_dff_2026-07-10.csv. Refs #712
inureyes
added
type:performance
…addendum The 2026-07-10 re-validation addendum justified the ~4% crossover shift as within noise by citing a "21.33-21.85 run to run" fallback spread attributed to the 2026-07-03 sweep. That range is actually this sweep's own llama-4-scout fallback (21.85/21.64/21.33 in cuda_gb10_issue712_fused_moe_dff_2026-07-10.csv); the 2026-07-03 sweep's fallback at Dff 8192 was 21.23/21.23/21.28, and the noise it flagged was on the fused side (one of three fused runs matching the fallback). The clause also called the 2.4% spread "larger than the shift itself" while the shift is stated as ~4%, which is a contradiction. Reattribute the 21.33-21.85 spread to the current sweep, keep the accurate statement that the 2026-07-03 sweep flagged this Dff point as noisy on the fused side, and change "larger than" to "comparable to" so the magnitude claim matches the ~4% shift. Conclusion is unchanged: 8192 stays confirmed. Data-only correction; no code touched. Refs #712
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Re-validates the CUDA fused-MoE Dff cap sweep behind
FUSED_MOE_MAX_DFF_CUDAon GB10 under the current MLX 0.32.1 pin, closing the "pending re-validation" note left by issue #330 / PR #711 in docs/benchmark_results/fused-moe-decode-kernel-design.md.Sweep table
Hardware: GB10 (DGX Spark), sm_121, CUDA 13.0. MLX pin:
57c66cac7cb3e5b1eb350488a61f1506b40d39f8(0.32.1, #703/#704). Harness:mlxcel-bench-decode, prompt "Hello, how are you today?", 100 decode tokens after a 20-token warmup, median of 3 runs per side,MLXCEL_FUSED_MOE_MAX_DFF=1(gather_qmm fallback) vs=20000(fused).The ratio=1.0 crossover interpolates to ~7672 between the phi-3.5-moe and llama-4-scout points, close to the prior ~8000 measured on MLX pin e9463bb. The ~4% shift sits within run-to-run noise at this Dff point: the 2026-07-03 sweep already flagged Dff 8192 as noisy (one of its three fused runs matched the fallback), and this 2026-07-10 sweep's own llama-4-scout fallback spans 21.33-21.85 tok/s run to run, a 2.4% spread comparable to the shift.
What changed
FUSED_MOE_MAX_DFF_CUDAinsrc/models/switch_layers.rsis confirmed at 8192, not re-tuned; this PR does not touch that file.MLXCEL_FUSED_MOE_MAX_DFFenv-var table row to state the confirmed result instead of "pending".benchmarks/cuda_gb10_issue712_fused_moe_dff_2026-07-10.csv.Test plan
cargo check --features cuda --lib --tests(pinned 1.93.1 toolchain) - clean,Finished dev profilewith no errors or warnings.cargo test --features cuda --lib switch_layers- 8 passed, 0 failed (unchanged since the constant was not modified).mlxcel-bench-decoderuns (4 models x 2 arms x 3 repeats), one process per run, rc=0 on every run.Closes #712