feat(sc): stream PPO actor and critic minibatches

[tianyi-zhang-02]

What does this PR do?

Lets Single Controller PPO stream actor and Megatron critic minibatches from native TQ replay instead of materializing one merged training payload.

The critic now has an explicit begin/train-chunk/finish/abort lifecycle. Chunks accumulate gradients and correctly normalized diagnostics, then take one optimizer step at the end of each critic epoch. The controller retains replay rows across critic and actor epochs, cleans up partial failures, skips invalid chunks, and rejects drop budgets that cannot preserve data-parallel batch divisibility.

Relates to #2625.

Validation

Current head 1253e0264a5a7665440fcff6607f63cd450164bc, rebased on upstream main at d633032b2a8017c69ddfff9183deb42cab6b36f4.

Check	Result
Current-head PPO/value/setup/SC actor suite on macOS arm64 (Python 3.12.2, pytest 7.4.4, torch 2.8.0 CPU, Ray 2.51.1; NVTX push/pop replaced with test-process no-ops)	233 passed, 8 skipped (7 GPU-gated; 1 missing optional megatron.bridge)
Ruff check, format check, diff check, and DCO	passed
Split critic parity on 2×H100 NVL, Megatron value model (2a9b108a pre-refresh head)	loss, grad norm, and every diagnostic metric matched
SC PPO functional + resume on 2×H100 NVL, Qwen2.5-0.5B, async vLLM, Megatron policy/value (same pre-refresh head)	steps 0–2 and resume 2–4 passed
Checkpoint and TensorBoard assertions (same H100 run)	checkpoints complete at steps 1–4; 10/10 assertions passed

The main refresh composes the intervening MOPD, NeMo Gym, refit, token-capture, training-claim, finalizer-metric, and rollout-checkpoint changes with PPO's multi-epoch row retention. The #3925 reconciliation uses the new sample_clears checkpoint-barrier mutation category for both PPO and non-PPO cleanup, while retaining the PPO-specific early cleanup on failure.

The 2-GPU Megatron split-parity and functional runs have not been rerun at the current head, so current-head GPU confirmation remains pending.

Functional entry point:

uv run --locked bash tests/functional/ppo_async_single_controller.sh

Before your PR is ready for review

• Read and followed the contributor guidelines
• Added the necessary tests
• Ran unit and functional tests
• Updated the Single Controller guide and example config

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[tianyi-zhang-02]

Closing to avoid competing with the active SingleController PPO roadmap in #4244 and #4256, which now split streaming policy updates and rollout/critic overlap into explicit designs. I would rather follow those implementations than keep a parallel train-pump design open.