Yifan Zhang*, Yifeng Liu*, Huizhuo Yuan, Zhen Qin, Yang Yuan, Quanquan Gu, Andrew C Yao
NeurIPS 2025 Spotlight
* Equal contribution · † Corresponding author
The question
What would it take to automate AI research itself?
01RL that scales: today's policy gradients are a brittle patchwork of heuristics.
02Models that hold long horizons: today's attention is memory-bound, and learning stops after pretraining.
03A loop that closes: model and harness must coevolve for improvements to compound.
My research makes each piece principled, and ships the result.
Pillar I
Agentic RL
Derive the objective, then scale agentic RL to long horizons.
Pillar I · two routes
REINFORCE vs. Online Mirror Descent
Route 1 · REINFORCEFollow the reward gradientPolicy gradient on the scalar reward, with no critic, no KL term and no ratio clipping. Stale rollouts are corrected by importance ratios and a trajectory gate.
Route 2 · Online Mirror DescentTake a KL-proximal stepEach update trades the learning signal against a KL penalty to the current policy. The KL direction, its form and the loss estimator are design choices.
FlashREINFORCE takes the first route; RPG, GPO and score centering take the second.
Pillar I · FlashREINFORCE
Single-rollout asynchronous REINFORCE
01One fresh batch. One rollout per prompt. Center rewards across independent prompts, then take one update and discard the batch. No learned critic.
02Ratio + trajectory gate. Use recorded sampler probabilities for token importance ratios; screen complete trajectories by a divergence statistic. No ratio clipping.
03Sample-mean optimization. Average tokens within each trajectory, then average trajectories, so long failures do not dominate by length.
Pillar I · Prefill–Decode Kernel Mismatch · Yifan Zhang and NVIDIA Research
Same weights, two policies
01Prefill–decode kernel mismatch. The sampler’s cached decoding and the trainer’s parallel scoring assign different probabilities at identical weights, with no parameter lag.
02Measured gap. On a matched 8,192-token prompt, vLLM and PyTorch differ by 0.093 in sampled-token log-probability (95th percentile) for Qwen3-4B, and by 0.044 for the hybrid Qwen3.5-4B.
03Attention vs. recurrence. Batch-invariant arithmetic aligns attention, whose cache stores inputs. Recurrent layers round the state at different points, so it does not align them.
Pillar I · Prefill–Decode Kernel Mismatch · Yifan Zhang and NVIDIA Research
Chunk-aligned decoding
01Match the schedule, not just the precision. The gap comes from where the recurrent state is rounded, so an FP32 state does not remove it.
02Keep what the trainer reads. The decoder keeps the last chunk-boundary state and the open chunk’s inputs, so each token comes from the trainer’s chunk kernel.
03Bit-exact in vLLM. Cached decoding reproduces the full prefill bit for bit on all 24 traces.
Pillar I · Prefill–Decode Kernel Mismatch · Yifan Zhang and NVIDIA Research
Aligned kernels keep accuracy and remove the kernel term
Chunk-aligned sampling, base Qwen3.5-4B. AIME avg@32 macro accuracy is 0.506, against 0.499 tokenwise; every 95% interval of the difference includes zero.Batch-invariant kernels in asynchronous RL, R1-Distill-1.5B at lag eight. Sampled forward KL averages 3.7×10⁻⁴ aligned and 5.4×10⁻⁴ ordinary over the first 3,600 updates.
→Bit-exact logits. Over about 40 million response tokens, the aligned arm’s trainer-to-sampler log-ratio has mean −3.4×10⁻⁷ and largest magnitude 6.2×10⁻⁶; the ordinary twin reaches 0.085 at the 95th percentile.
02But GRPO / PPO are a patchwork of clipping and KL hacks — unstable, hard to reason about.
Start from the KL-regularized objective — and be exact.
ICLR 2026
KL-regularized policy gradient, done right
A unified design space for KL-regularized policy-gradient algorithms — with the correct gradient and stop-gradient treatment.
ImpactAdopted in DeepSeek-V3.2.
→More stable training; recovers and repairs popular methods as special cases.
Pillar I · RPG · ICLR 2026
The exact gradient for every KL variant
RPG-REINFORCE with RPG-Style Clip gains up to +6 points over DAPO on AIME24 and AIME25.
01One derivation covers forward and reverse KL, normalized and unnormalized; the widely used k3 penalty is exactly the unnormalized KL.
02Stop-gradient done right. Conditions under which REINFORCE-style losses match fully differentiable surrogates in gradient.
03Repairs GRPO. Its KL term has an off-policy importance-weighting mismatch; RPG corrects it.
04RPG-Style Clip. Clipped importance sampling keeps off-policy training stable at scale.
Adopted in DeepSeek-V3.2 · Curves: AIME24 and AIME25 mean@32 accuracy during training, 4k context length.
Pillar I · GPM & GPO · ICML 2025
Beyond Bradley–Terry preferences
Preference embedding places each response in a latent space, so the model can represent cyclic, intransitive human preferences that a scalar reward cannot.
→GPM embeds each response once (linear queries, not $\mathcal{O}(K^2)$) and consistently outperforms the Bradley–Terry reward model on RewardBench.
→GPO generalizes reward-based RLHF, and each iteration is an online mirror descent step: $\pi_{t+1}(y\mid x)\propto\pi_t(y\mid x)\exp\!\big(\eta\,s(y\succ\pi_t\mid x)\big)$.
→At the token level with $\beta=0$, GPO reduces to score centering.
Pillar I · GPM & GPO · experiments
GPM models what Bradley–Terry cannot
Setting
BT reward model
GPM
Gain
Cyclic preferences · dataset No. 1
62.4%
100.0%
+37.6
Cyclic preferences · dataset No. 3
50.0%
100.0%
+50.0
RewardBench average · Gemma-2B-it
74.85
82.29
+7.44
RewardBench average · Llama-3.1-8B-Instruct
90.56
91.90
+1.34
→Cyclic preferences: GPM reaches 100% where the BT model stays near the 50% random-guess level.
→RewardBench: the largest gains are in Chat and Chat-Hard, 67.32 → 79.61 and 63.37 → 75.66 with Gemma-2B-it.
→GPO: models aligned with GPO reach higher AlpacaEval 2.0 win rates than BT-based alignment.
GPM & GPO · arXiv:2410.02197 ↗ · ICML 2025 · Cyclic rows are accuracy; GPM embedding dimension is 6 for Gemma-2B-it and 8 for Llama-3.1-8B-Instruct.
2. Fit this equation by least squares on the sampler’s tokens. The unknown $c(s)$ is replaced by $k_\theta(s)=\mathrm{KL}(\mu\,\|\,\pi_\theta)$, which gives the residual $\delta_u$ zero mean under $\mu$:
3. Differentiate: $\nabla_\theta\delta_u=-\tilde g_\theta$, the score centered under the sampler. Since $\mathbb{E}_{\mu}\tilde g_\theta=0$, state-only terms drop and the return $R$ replaces $A^{\mu}$:
01One transition. Prompt, response, user and tool tokens all take the same update; the prompt–response boundary does not reset state or caches.
02Exact replay. After each parameter update, the trainer rebuilds encoder KV, recurrent states and decoder caches under the current parameters.
03Fixed behavior probabilities. The sampler’s recorded log-probabilities stay as the ratio denominator.
04Same execution settings. Replay uses the rollout’s chunk size, masks and positions.
RLT · arXiv:2610.07591 ↗ · Replay rebuilds current-policy state and caches; the recorded sampler probabilities remain fixed.
Pillar II · RLT · hardware efficiency
Hardware-efficient: parallel wherever tokens are known
01Parallel encoder. Known tokens build the causal global KV memory together.
02Chunked feedback (RLT-2). One feedback state per chunk of $B$ tokens cuts sequential decoder stages from $TL_D$ to $\lceil T/B\rceil L_D$, with the same parameters.
03Batched across sequences. Independent sequences share one batched kernel for their next decoder update.
04$B$ trades depth for parallelism. $B=1$ gives the longest recurrent path; a larger $B$ puts more known tokens in each parallel step.
RLT · arXiv:2610.07591 ↗ · The interval is a tradeoff: at $B=4$, 64-bit parity stays at 98.99% while S₅ swaps at 64 operations fall from 100% to 19.60%.
Pillar III
AutoResearch
Agents that automate research itself.
Pillar III · Agora · collective AutoResearch
Git records the research process
01Append-only contributions. Code, hypotheses, results and verifications form a dependency graph that workers can check out and reproduce.
02Evidence guides attention. Reuse and verification propagate scores; analyze() exposes frontier results, contested claims and underexplored branches.
Agora · arXiv:2609.18094 ↗ · NeurIPS 2026 AutoMLR Workshop Oral · NVIDIA Technical Report Workers choose experiments from shared state; there is no central planner assigning tasks.
Pillar III · Agora · sustained research run
Collective search leaves reproducible evidence
141 donors, 32 architecture families. Transfer into a 119.6M-parameter attention–SSM hybrid using donor weights and forward passes.
3.3923 → 1.899 bpb. The development evaluator scores 200 FineWeb-Edu texts; pretraining and fine-tuning are prohibited.