fix(policy): surface the uncovered-rank error during HTTP weight refit

[tianyi-zhang-02]

What does this PR do?

Makes the existing uncovered-rank error in stream_weights_via_http_impl reachable.

The function currently reads cached IPC topology before resolving the rollout-engine URL, so an uncovered rank raises KeyError: 'gather_src'. This moves those reads below URL resolution, allowing the intended RuntimeError to report the bad rank mapping.

Covered-rank behavior is unchanged.

Testing

• Added uncovered-rank and covered-rank regression tests: 2 passed locally
• ruff check, ruff format, and pyrefly pass
• Three existing ZMQ IPC tests fail identically on main and this branch on Windows because ipc:// is unsupported

Issues

Found while investigating #3288. Draft PRs #3330 and #3426 also touch this function.

cc @RayenTian @yuki-97

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[yuki-97]

@bg51717 to review

[bg51717]

LGTM. This change ensures uncovered policy ranks raise the intended RuntimeError instead of a KeyError, with tests covering both covered and uncovered rank paths.

[bg51717]

Merged the latest upstream/main into this branch and removed the redundant explanatory comment.

[bg51717]

/ok to test 78f147d

[tianyi-zhang-02]

Two things on the red checks.

L0_Unit_Tests_Sglang was flaky, not a real failure. That run ended with

=========== 79 passed, 10 skipped, 61 warnings in 1026.58s (0:17:06) ===========
./tests/run_unit.sh: line 44: 20394 Segmentation fault (core dumped) pytest $pytest_args

All 79 passed. The segfault is at interpreter shutdown, after the summary line. The job is judged by tail -n 1 err.log | grep "Finished successfully.", and that never gets written when the process dies on a signal, so the whole shard goes red. CI quality check is just the roll-up of it. Nothing here runs at shutdown — the PR moves two reads inside stream_weights_via_http_impl.

I also brought the branch up to date. Should be clean merge now.

Needs a fresh /ok to test when convenient :)

[tianyi-zhang-02]

Closing this — main fixed it, and better than I did.

stream_weights_via_http_impl was rewritten since I opened this. It now reads the topology with .get() instead of [...], and an uncovered rank takes an explicit placeholder branch that drains the bucket iterator and returns. That's the right answer: the iterator lazily drives export_hf_weights, whose PP broadcast and TP gather are collective over the whole trainer world, so raising there — which is what this PR did — would have hung the other ranks.

Thanks for the review @bg51717 and @yuki-97, sorry it went stale. :)