feat(research): add native turn-level credit assignment

[nvegesna-netizen]

What does this PR do?

Adds a self-contained research/turn_level_credit project that preserves
native per-turn environment rewards and composes token-aligned auxiliary
credit with NeMo-RL's existing trajectory-level GRPO advantage.

This follows the research-incubation direction discussed on #1606. It does
not change files under nemo_rl/, does not claim a task-quality improvement,
and does not implement TRACE. The project supplies the reusable plumbing and
strict validation needed for controlled long-horizon experiments.

The implementation:

• captures raw reward, reward components, and terminal state at each native
  environment transition;
• tensorizes compact turn rewards and assistant-token spans before message
  flattening;
• validates turn counts, terminal boundaries, dtypes, devices, masks, spans,
  and equality between raw turn sums and raw trajectory reward;
• wraps the existing GRPO estimator, preserving strict base-object identity
  when turn_weight=0;
• supports immediate and discounted return-to-go auxiliary credit;
• fails at startup outside the explicitly validated synchronous native-GRPO
  path; and
• restores all scoped runtime hooks after training.

Issues

Related to #1326. Supersedes neither #1606 nor NVIDIA-NeMo/Gym#1298; the
latter remains the coordination point for a versioned NeMo Gym step-reward
contract.

Usage

cd research/turn_level_credit
uv run run_grpo_turn_credit.py \
  --config configs/grpo_math_0.5b_turn_credit.yaml

Strict macro-only control with trace plumbing enabled:

uv run run_grpo_turn_credit.py \
  --config configs/grpo_math_0.5b_turn_credit.yaml \
  turn_credit.turn_weight=0

Multi-turn pilot result

The included 2x2 sliding-puzzle recipe was evaluated with three matched seeds,
ten optimizer steps per arm, and 32 final validation rollouts per seed. The
control used turn_weight=0; the treatment used immediate environment credit
with turn_weight=0.2.

Seed	Control final	Treatment final	Difference
42	0.21875	0.28125	+0.06250
43	0.25000	0.15625	-0.09375
44	0.31250	0.31250	0.00000

The mean seed-level difference was -0.010417, with a 95% t interval of
[-0.205779, 0.184946]. The fully paired final outcomes retained for seeds
43-44 also showed no significant difference (exact McNemar p=0.663624).

This establishes that the mechanism runs in a genuine multi-turn training
loop, but it does not establish a task-quality gain. The environment returns
zero reward until the puzzle is solved, so this pilot primarily tests
terminal-turn emphasis rather than dense process credit. No default credit
weight is recommended from these results.

Validation

• All 54 research unit/integration tests passed in a Linux GPU environment
  matching NeMo-RL's nightly dependencies.
• Ruff check and format check passed for all research files.
• Pyrefly reports zero errors across all seven turn-credit source modules.
• Shell syntax and the three root recipe/test-suite invariants passed.
• Full Sphinx HTML documentation passed with warnings treated as errors.
• uv lock --check resolved all 549 packages against the recorded submodules.
• All eight PR commits are DCO signed off.
• An exact-source one-GPU treatment with turn_weight=0.2 passed all preflight
  checks, activated native turn-level credit, completed one optimizer step,
  and reached the configured max-step boundary.
• The matched zero-weight control used the same source, container, payload,
  configuration, and preflight; its only semantic experiment difference was
  turn_weight=0.0. It also completed one optimizer step successfully.

Limitations

The bundled one-turn math recipe validates engineering plumbing only. A task
quality claim requires a genuinely long-horizon environment and controlled
outcome-only, zero-weight, immediate-credit, and return-to-go comparisons with
matched prompts, seeds, model, and compute budget.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.