Add exact dense Qwen programs and shared operator improvements by kiddyboots216 · Pull Request #43 · togethercomputer/xorl

kiddyboots216 · 2026-08-10T05:17:14Z

Summary

Add capability-resolved exact numerical programs for dense Qwen3 and hybrid-GDN dense Qwen3.5 training.
Reinstall the exact dense-Qwen families-v2 and batch-invariant trunk program after TP unfusing and LoRA projection replacement, before FSDP; unsupported QLoRA projection types fail closed.
Preserve configured nonzero LoRA-B startup bytes in the registered default-adapter slots instead of replacing them with zeros.
Keep exact GLM active-LoRA rank/alpha construction-scoped and reject incompatible heterogeneous session specs at registration.
Synchronize single-writer full-weight save failures across ranks before the completion barrier.
Use the shared one-round FP32 SwiGLU program for admitted dense/shared MLP sites; routed experts remain inside their architecture-owned fused kernels.
Route families-v2 RMSNorm callers across dense Qwen, Qwen3.5/3.6, and GLM while keeping all shipped residual hidden sizes, including 6144, on the intended fused realization.
Select Class-B RoPE, batch-invariant trunk matrix products, matching LM-head arithmetic, and selected-token loss through architecture-owned contracts.

Stack

Base: Update XoRL submodule branches and SGLang pin #41 at 762205400ad0b136817c3a3f6658f39500d2ded1
Companion serving: Add exact dense Qwen programs and shared operator improvements xorl-sglang#15 at 1fc3396aa122e0f8eb7b1ea4fe0051250cf1bc3c
Head: f1238691852d5436541ec8ea0bcb8d5a018c3752
The trainer gitlink pins that exact companion-serving object.

Validation

Post-restack focused lifecycle, dense-Qwen contract, exact-GLM rank, nonzero-B, and save-failure suite: 106 passed, 3 skipped.
Targeted ruff check: passed.
git diff --check and recursive submodule cleanliness: passed.
git range-diff preserves all 12 live payload commits plus the lifecycle repair; only the obsolete intermediate sampler-pin delta collapses into the corrected Update XoRL submodule branches and SGLang pin #41 base.

broly-code-security-scanner · 2026-08-10T05:17:25Z

Broly Security Scan

Note

✅ Clean scan
No vulnerabilities detected in this PR.

Note

Re-scan this PR anytime with /broly scan — useful after /broly undismiss, or to refresh findings without a new push.

Broly — SAST (zai-org/GLM-5.2) · Secrets · SCA · IaC · GH Actions · Base Images · Supply Chain Threats · Exploit Chains · Adversarial Verification

We're continuously improving Broly's accuracy and finding quality — your feedback is valuable. False positives, missed findings, bugs, and feature requests all welcome.

The cherry-picked #43-chain change makes fused_silu_and_mul one-round for EVERY caller (Qwen2/3, Llama3, OLMo2, DeepSeek-V3, GLM4/5, MoE defaults) — review measured 53,619/196,608 BF16 outputs changed on non-exact paths. An unblocking branch must not silently change non-exact model numerics, so: - src/xorl/ops/fused_silu_and_mul.py: the pre-landing TWO-ROUND module content is restored verbatim as the default program; the one-round FP32 program is ADDITIVE as exact_fp32_silu_and_mul (Hopper Triton kernel + differentiable eager fallback + analytic-backward autograd Function), byte-paired with serving's fp32_silu_and_mul. - qwen3_5 / qwen3_5_moe MLPs dispatch exact_fp32_silu_and_mul ONLY under _qwen35_exact_contract; the qwen3_5_moe serving-value shared-expert site (exact-lane-only code) uses it directly. - qwen3, qwen3_moe, glm5 modeling and the three GLM52 exact tests revert to da77725 byte-for-byte: zero change off the exact path. - tests/ops/test_fused_silu_and_mul_scoping.py pins both sides: default op == two-round reference bytes (and != one-round: discriminating), non-exact Qwen3_5MLP == pre-landing composition bytes, exact Qwen3_5MLP == one-round. Deliberate divergence from the #43 chain end-state (universal one-round), documented in the module docstring and LANDING.md; the chain supersedes this split when it lands.

(cherry picked from commit 8a97bcb, origin/feature/qwen3-dense-exact-kernels-20260810, PR #43 chain — the trainer half of serving-side xorl-sglang f10b907d8, which reached cleanup-tests alone via pin advance d13788d and broke trainer<->sampler byte parity on the dense Qwen3.5-0.8B exact pair; hunt record: results/k3_first_divergence/LANE_LOG.md on branch k3-first-divergence) Adaptations for cleanup-tests (da77725): - src/xorl/ops/fused_silu_and_mul.py: context-anchoring conflicts only (cleanup-tests never modified this file since the chain fork); resolved to the 8a97bcb post-image verbatim. - tests/models/test_qwen35_exact_swiglu_policy.py: delete/modify conflict (file exists only on the #43 chain); took the 8a97bcb post-image. - Follow-up 32f6e6c 'Reconcile exact SwiGLU and RMSNorm dispatch gates' is NOT included: its test deletion targets a #43-chain-only file absent here, and its bi_families_v2.py retune presupposes the chain's earlier 4-arg _v2_norm_use_split (Hopper RMSNorm trainer work). That dispatch gate is perf-only per its own contract docstring, and byte-neutrality of the RMSNorm realizations is measured (16384/16384 pair equality at serving pin c4de30cba, which contains the serving-side Hopper routing 91c90570a). It lands with the full #43 chain.

8a97bcb's diff presupposes its chain parent: the two Qwen3.5 MLP __init__ hunks that force the fused (one-round FP32) SwiGLU path whenever _qwen35_exact_contract is set, overriding _activation_native. On the #43 chain they land in 55548e7 ('Add exact Qwen3-8B training numerics'); this commit extracts EXACTLY those two hunks (nothing else — the Qwen3-8B exact admission, bi_families_v2 additions, and the since-deleted two-round test stay with the chain). With them, both files are byte-identical to the 8a97bcb post-image, and tests/models/test_qwen35_exact_swiglu_policy.py passes. Without this, the exact Qwen3.5 lane keeps activation_native=True and the two-round eager SwiGLU, and the trainer<->sampler byte divergence persists (the policy test catches it: _use_fused_silu False under the exact config).

Restores the byte coverage deleted with the old two-round test (32f6e6c on the #43 chain), now asserting the intended end-state (user-confirmed): the DEFAULT fused_silu_and_mul == the one-round FP32 reference bytes and != the two-round reference (discriminating power), for exact and non-exact callers alike — a non-exact Qwen3_5MLP and the exact-contract Qwen3_5MLP both match the one-round composition bitwise. The non-exact byte change is deliberate, reviewed #43 intent; this gate pins it so a silent re-scoping or two-round regression fails loudly.

kiddyboots216 · 2026-08-13T21:01:41Z

Superseded by #57, which consolidates the complete exact trainer-serving stack directly onto main.

kiddyboots216 mentioned this pull request Aug 10, 2026

Add exact dense Qwen programs and shared operator improvements togethercomputer/xorl-sglang#15

Closed

kiddyboots216 changed the title ~~Add architecture-owned exact dense Qwen3 kernels~~ Add capability-resolved exact dense Qwen3 kernels Aug 10, 2026

kiddyboots216 changed the title ~~Add capability-resolved exact dense Qwen3 kernels~~ Add capability-resolved exact dense Qwen kernels Aug 10, 2026

kiddyboots216 changed the title ~~Add capability-resolved exact dense Qwen kernels~~ Add exact dense Qwen programs and shared operator improvements Aug 10, 2026

kiddyboots216 force-pushed the feature/qwen3-dense-exact-kernels-20260810 branch from 6da499e to e5f91b1 Compare August 11, 2026 18:01

kiddyboots216 changed the base branch from main to chore/update-xorl-submodules August 11, 2026 18:02

This was referenced Aug 11, 2026

Update XoRL submodule branches and SGLang pin #41

Closed

Unify canonical MoE folding across Qwen and GLM #44

Closed

Unify DSV4 onto the canonical MoE fold and requalify to K3 = 0 #48

Closed

kiddyboots216 added 13 commits August 12, 2026 02:03

Add exact Qwen3-8B training numerics

55548e7

Route Hopper RMSNorm away from spilling kernels

3de6933

Admit exact dense Qwen3 by capability

0f1e714

Admit exact dense Qwen3.5 by capability

59c400e

Cover shared exact operators across model families

2455f10

Use one-round FP32 SwiGLU in exact MLPs

8a97bcb

Fix single-writer sampler weight export

93a0fc6

Format exact dense and shared MLP tests

33c2931

Reconcile exact SwiGLU and RMSNorm dispatch gates

32f6e6c

feat(glm52): qualify exact rank-16 LoRA

fd479ed

fix glm52 rank16 exact lora admission

8ba250e

fix(glm52): generalize exact LoRA rank admission

0d00bf5

fix exact trainer lifecycle contracts

f123869

kiddyboots216 force-pushed the feature/qwen3-dense-exact-kernels-20260810 branch from 61d8222 to f123869 Compare August 12, 2026 02:05

kiddyboots216 added 4 commits August 12, 2026 02:06

Pin admitted GLM exact KV capacity

1043952

Advance exact Qwen sampler pin after CI repair

a288f00

Defer nonzero LoRA-B initialization past DCP restore

f83e22b

Advance sampler for normal GLM LoRA batches

2a0d78e

kiddyboots216 requested a review from qywu August 12, 2026 04:11

This was referenced Aug 13, 2026

Audit LoRA projection coverage across model families #49

Closed

Scope one-round FP32 SwiGLU to the exact Qwen3.5 contract #51

Closed

kiddyboots216 closed this Aug 13, 2026

Navigation Menu

Uh oh!

Add exact dense Qwen programs and shared operator improvements - #43

Add exact dense Qwen programs and shared operator improvements#43
kiddyboots216 wants to merge 17 commits into
chore/update-xorl-submodulesfrom
feature/qwen3-dense-exact-kernels-20260810

kiddyboots216 commented Aug 10, 2026 •

edited

Loading

Uh oh!

broly-code-security-scanner Bot commented Aug 10, 2026

Uh oh!

kiddyboots216 commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

kiddyboots216 commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Stack

Validation

Uh oh!

broly-code-security-scanner Bot commented Aug 10, 2026

Broly Security Scan

Uh oh!

kiddyboots216 commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

kiddyboots216 commented Aug 10, 2026 •

edited

Loading