NETEASE · RESEARCH TALK

Flash
REINFORCE

Critic-Free Single-Rollout Asynchronous RL
for Agentic Language Models

Yifan ZhangNVIDIA · NetEase Research Talk

Press → or Space to begin · ? for help

RESEARCH QUESTION

Learning from each completed interaction

Can we train an agent reliably with
one rollout per prompt,
even when trajectories arrive asynchronously?

The learning rule must handle missing sibling baselines and stale histories.

FlashREINFORCE · Introduction

MOTIVATION

Agent trajectories finish at different times

Group-relative learning

A prompt-local baseline requires sibling rewards.

Asynchronous learning

Workers can submit each completed trajectory independently.

Illustrative schedule, not measured runtime data

MOTIVATION

Rollout budget also determines coverage

32 prompts × 4 rollouts

Repeated samples support within-prompt comparison.

128 prompts × 1 rollout

The same 128 trajectories cover four times as many prompts.

Fixed budget: B = 128 trajectories per update

Prompt coverage and baseline quality compete at a fixed sampling budget.

FlashREINFORCE · Introduction; Appendix, group–batch comparison

CHALLENGES

What changes with a single rollout?

01
No sibling baselineWe need useful feedback across independent prompts.
02
Stale behavior historiesAction correction alone leaves history-distribution mismatch.
03
Coarse trajectory rewardsLong failures can dominate updates and penalize useful steps.

FlashREINFORCE · Introduction

SETUP

An asynchronous, one-pass learner

ROLLOUT WORKERS

Policy snapshots μᵢ

One trajectory per prompt
Store sampled-token log μᵢ

COMPLETION QUEUE

Next B completions

Mixed behavior snapshots
No sibling-group wait

LEARNER πθ

One update

Center, screen, average
Discard the used batch

“Fresh” means not previously used for an update. A fresh trajectory can still be stale.

ρᵢ,ₜ = exp(log π_θ(aᵢ,ₜ ∣ hᵢ,ₜ) − log μᵢ(aᵢ,ₜ ∣ hᵢ,ₜ))

FlashREINFORCE · Setup and the Local Off-Policy Objective

ANALYSIS

A local off-policy surrogate

ℒ_μ(π) = J(μ) + ∑ₜ E_{h ∼ dᵗμ} E_{a ∼ π(·∣h)} [Aᵗμ(h,a)]

Behavior quantities stay fixed

Histories follow dᵗμ.
Continuation values come from μ.

The action distribution changes

The inner expectation uses π.
At π = μ, the local gradient matches policy gradient.

J(π) = E_τ∼π[R(τ)] denotes expected return.

FlashREINFORCE · Local surrogate; Appendix Proposition: Exact local gradient

ANALYSIS

What token importance sampling corrects

∇_θℒ_μ(π_θ) = E_τ∼μ [∑ₜ ρₜ R(τ) ∇_θ log π_θ(aₜ ∣ hₜ)]

Exact conditional correction

The ratio corrects the action distribution at each stored history.

Residual history mismatch

Stored histories still follow μ.
The target return depends on histories from πθ.

Prefix or sequence ratios can address occupancy, with potentially high variance.

FlashREINFORCE · Local-gradient identity

ANALYSIS

Surrogate error accumulates with drift

∣J(π) − ℒ_μ(π)∣ ≤ 2 A_max(∑ₜ εₜ)² ≤ A_maxT² κ̄

01
εₜ is the history-wise supremum of total variation at position t.
02
κ̄ averages history-wise supremum KL over a fixed horizon T.
03
The bound motivates sequence-wide screening; it does not certify the implemented gate.

Assumptions: bounded behavior advantages, common environment and prompt distribution, fixed horizon with absorbing padding.

FlashREINFORCE · Accumulated-drift proposition; coupling argument also reviewed in TRM

METHOD

FlashREINFORCE

01 · One-batch REINFORCE

Center rewards across the next batch of completed trajectories.

02 · Sequence trust

Retain token IS and screen each complete trajectory for drift.

03 · Sample mean

Average within each trajectory before averaging across the batch.

One rollout per prompt. One optimizer step per fresh batch. No critic.

FlashREINFORCE · Method overview

METHOD · 01

Signed feedback without sibling rollouts

R̄ = 1B∑ⱼ Rⱼ　　Aᵢ = Rᵢ − R̄

Binary reward example	Rᵢ	Aᵢ when R̄ = 0.8
Success	1	+0.2
Failure	0	−0.8

For general scalar rewards, a negative Aᵢ means below-batch-mean return.

Mixed-reward batches give failures a signed learning signal.

FlashREINFORCE · One-batch REINFORCE; example values are illustrative

METHOD · 02

A low-cost sequence drift screen

dᵢ,ₜ = p log(p/q) + (1 − p) log((1 − p)/(1 − q))

p = μᵢ(aᵢ,ₜ ∣ hᵢ,ₜ),　q = πθ(aᵢ,ₜ ∣ hᵢ,ₜ)

D̄ᵢ = 1Tᵢ ∑ₜ dᵢ,ₜ　　mᵢ = 𝟙[D̄ᵢ ≤ δ]

D̄ᵢ ≤ mean full KL: exceeding δ flags drift; passing does not certify closeness.

FlashREINFORCE · Sequence trust; Appendix, projected-trust relation

METHOD · 03

Trajectory length changes update weight

Reduction	Weight on mean trajectory loss Lᵢ	Effect
Sample mean	1 / B	Equal explicit trajectory weight
Token mean	Tᵢ / ∑ⱼ Tⱼ	More weight on longer trajectories

Why normalize?

Long failures contain useful steps as well as errors. Coarse negative feedback can grow with length.

What remains unresolved?

Averaging changes length weighting. It does not identify which tokens caused success or failure.

FlashREINFORCE · Sample-mean optimization; Appendix loss reductions

METHOD

The complete update

ĝ = 1B ∑ᵢ mᵢ AᵢTᵢ ∑ₜ ρᵢ,ₜ ∇_θ log π_θ(aᵢ,ₜ ∣ hᵢ,ₜ)

01
Aᵢ supplies batch-centered signed feedback.
02
mᵢ applies one admission decision to the complete trajectory.
03
ρᵢ,ₜ corrects conditional actions; 1/Tᵢ controls length weighting.

Aᵢ, mᵢ and behavior probabilities are held fixed during the update.

FlashREINFORCE · Algorithm 1 and practical gradient

IMPLEMENTATION

One fresh batch, one optimization step

01
CollectUse the next B completed trajectories and their actual behavior log-probabilities.
02
ComputeCenter rewards, compute token ratios, and make one gate decision per trajectory.
03
Update and discardTake one full-batch optimizer step, then use new completions.

Numerical guard: float32 log-ratio, clamped to [−30, 30] before exponentiation.

FlashREINFORCE · Algorithm 1; Appendix implementation details

EXPERIMENTS

Reasoning, tool use, and interaction

Setting	Model	Evaluation
Long CoT	R1-Distill-Qwen-1.5B	AIME24/25, avg@32
Math reasoning	Qwen2.5-Math-1.5B	Five benchmarks, avg@16
Python tools	Qwen2.5-7B / Qwen3-30B-A3B	AMC23, Minerva, AIME25
Interactive tasks	Qwen2.5-7B-Instruct	ALFWorld seen / unseen

Benchmark means weight tasks equally. avg@k is mean per-sample accuracy, not pass@k.

FlashREINFORCE · Evaluation protocols; Appendix training settings

RESULT · MATHEMATICAL REASONING

Higher mean accuracy with half the rollouts

Qwen2.5-Math-1.5B	Group size	Rollouts	Five-task mean (%)
Base model	—	0	19.5
C-RF + NTF (published)	1	512k	35.3
GRPO (published)	16	512k	36.3
FlashREINFORCE	1	256k	38.0

FlashREINFORCE: peak-mean checkpoint at step 2,000.
Published baselines use synchronous training and 1,000 updates.

+1.7 percentage points over reported GRPO; not a matched wall-clock study.

FlashREINFORCE · Table: Five-benchmark comparison; published baselines from Wu et al., C-RF + NTF

RESULT · LONG COT

Stable training through 6,000 updates

R1-Distill-Qwen-1.5B, policy lag ≈ 4. Raw avg@32 evaluations; line segments connect checkpoints.

Final reported means: 33.7 with trust, 30.6 without trust.

Source: assets/r1_lag4_gate_aime.csv and r1_lag4_nogate_aime.csv

RESULT · LONG COT

Length decreases as entropy settles

With sequence trust. Each point averages a consecutive 200-update window.

Stable long-run behavior accompanies the accuracy improvement.

Source: assets/r1_lag4_gate_dynamics.csv

RESULT · PYTHON TOOLS

Tool use persists during training

Qwen2.5-7B, step 600	AMC23	Minerva	AIME25	Mean	Calls
GRPO	51.2	32.3	7.5	30.3	0.00
FlashREINFORCE	60.6	28.8	21.7	37.0	3.25

GRPO stops calling the tool within roughly 200 steps.
FlashREINFORCE retains multi-turn tool use.

Matched steps and rollout budgets. Calls are averaged over the 100 updates ending at the checkpoint.

FlashREINFORCE · Python-tool main results, panel (a)

RESULT · 30B MOE

Scaling to Qwen3-30B-A3B at lag 8

Step 800, 102.4k rollouts	Lag	AMC23	Minerva	AIME25	Mean
GRPO	≈1	93.8	40.4	46.7	60.3
FlashREINFORCE	≈8	95.6	44.9	60.8	67.1

+6.8pp

Three-task mean accuracy
at the same rollout budget

Flash: δ = 10⁻², no routing replay. Evaluation: avg@4; thinking disabled.

FlashREINFORCE · Python-tool main results, panel (b)

RESULT · 30B MOE

Sustained performance at policy lag 8

R3 uses routing replay and δ = 10⁻³. Raw avg@4 evaluations, joined without smoothing.

R3 mean: 63.7–66.8 over updates 1,000–1,450; final score 66.4.

Source: assets/qwen3_moe_tool_evals.csv

RESULT · INTERACTIVE TASKS

ALFWorld seen and unseen success

Method	Group size	Seen (%)	Unseen (%)
GRPO (published)	8	78.6	76.8
C-RF + NTF (published)	1	90.5	86.3
FlashREINFORCE	1	98.3	96.5

FlashREINFORCE uses 12.8k training trajectories
and evaluates at step 200.

C-RF also uses one rollout per prompt, so prompt coverage alone does not explain the difference.

FlashREINFORCE · ALFWorld results; published baselines from Wu et al.

ABLATIONS

Fresh updates and signed feedback matter

Freshness at equal budget

Schedule	Mean (%)
Fresh batch each step	28.45
4 sequential minibatches	26.80

Math: 800 steps, 102.4k trajectories.

Signed reward feedback

Learning signal	Mean (%)
Batch-centered	36.2
Positive-only	9.8

7B tools, step 700. Calls: 3.30 vs 0.00.

FlashREINFORCE · Fresh one-batch and signed-feedback ablations

ABLATIONS

Sample mean avoids the observed length collapse

Continuation from step 900	Reward	Length	Calls	Truncation
Starting checkpoint	0.430	2,573	3.87	18.8%
Sample mean, ≈150 steps	0.477	2,518	4.01	17.2%
Token mean, ≈150 steps	0.403	3,947	2.18	45.3%

Token-level penalties do not directly reward shorter responses.

Possible mechanism: coarse negative feedback weakens useful behavior. The specific causal chain remains unverified.

FlashREINFORCE · Loss-reduction continuation ablation

ABLATIONS

The granularity of admission matters

Math, same δ = 10⁻³	Last step	AMC23: peak / last	Outcome
Sequence-level	2,160	55.6 / 53.1	Stable
Token-local	1,237	51.9 / 26.3	Collapsed

Token-local masking

Deleting a token loss leaves later stored histories unchanged.

Sequence admission

Rejecting a trajectory removes all of its stored-history contributions.

The comparison changes admission granularity. It is not a gate-on versus gate-off experiment.

FlashREINFORCE · Trust-granularity ablation

ABLATIONS

Group and batch centering remain close

128 prompts × 1

Batch-mean baseline
Flash loss, token IS and gate

32 prompts × 4

Within-prompt mean baseline
No standard-deviation normalization

0.22pp

Late mean advantage for group centering
across seven paired evaluations after step 3,000

R1, policy lag 4, 128 trajectories per update. Curves remain broadly similar through approximately 4,000 updates.

FlashREINFORCE · Appendix, group–batch comparison

LIMITATIONS

What the evidence does not establish

01
A full trust-region guaranteeThe sampled-action proxy can miss drift; the practical update is masked and length-normalized.
02
General reward robustnessContinuous rewards and high-noise business settings need dedicated experiments.
03
Universal scaling behaviorBatch-size sweeps, longer environments and matched wall-clock studies remain open.

FlashREINFORCE · Limitations; scope of reported experiments

CONCLUSION

A practical recipe for asynchronous agent RL

01
Single-rollout learningBatch centering supplies signed feedback without a critic or sibling group.
02
Drift-aware updatesToken IS and sequence screening support sustained asynchronous training.
03
Trajectory-aware weightingSample mean limits the amplification of long failed trajectories.

Evidence spans long CoT, 30B MoE tool use, mathematical reasoning and ALFWorld.

FlashREINFORCE · Conclusion

RESOURCES

Paper, code, and related work

FlashREINFORCE paperMethod, main results, ablations and experimental appendix Project websiteResearch overview and additional resources NVIDIA-NeMo / labs-moltCode link listed in the author-named preprint Discussion indexPrepared questions and research discussion backups

Related directions: REINFORCE / RLOO, GRPO, C-RF + NTF, DPPO and Trust Region Masking.

Visual direction: yifzhang.com/slides

DISCUSSION

Questions & discussion

01Gate contribution 02Gate on / off 03Continuous rewards 04Advantage and difficulty 05Uniform-reward batches 06Noisy rewards 07Long trajectories 08Sparse rewards 09Lag and length collapse 10Rejected rollout cost 11Memory and approximation 12Training practice

Select a topic for backup material. Press Q to return to this page.

Q01 · DISCUSSION BACKUP

How much does sequence trust contribute?

R1 mean accuracy (%)	With gate	Without gate	Difference
Step 4,992	32.2	30.6	+1.6 pp
Around step 5,900	33.7	30.6	+3.1 pp

Final evaluation steps differ slightly: 5,896 versus 5,888.

Batch-size sensitivity has not been systematically evaluated.

Main talk: long-CoT results; Appendix gate-threshold experiments · Back to discussion

Q02 · DISCUSSION BACKUP

Is the gate necessary or a safeguard?

0.13%

Trajectory rejection in the reported
Math run with δ = 3×10⁻³

01
The R1 gate-only comparison supports a measurable long-run benefit.
02
Low rejection is consistent with outlier screening, but does not prove the causal mechanism.
03
ALFWorld removes both IS and trust; its drop cannot be assigned to the gate alone.

FlashREINFORCE · Trust ablations and threshold discussion · Back to discussion

Q03 · DISCUSSION BACKUP

Does centering extend to continuous rewards?

Continuous terminal score

Aᵢ = Rᵢ − R̄ remains defined.
Cross-prompt score calibration becomes important.

Dense stepwise rewards

Return-to-go can use intermediate feedback.
Total-reward centering alone does not assign step credit.

01
A prompt-local baseline adapts to difficulty, but its finite-sample estimate is noisy.
02
The paper does not report continuous-reward or high-noise RLHF experiments.

FlashREINFORCE · Reward centering; extensions are unvalidated · Back to discussion

Q04 · DISCUSSION BACKUP

Does a hard success get negative advantage?

Binary reward: Rᵢ = 1　⟹　Aᵢ = 1 − R̄ ≥ 0

Easy and hard successes receive the same scalar advantage within a batch.

01
Batch centering does not estimate prompt difficulty.
02
The controlled group–batch comparison is close, with group centering ahead by 0.22 pp late in training.

More data does not automatically remove bias from a chosen surrogate.

FlashREINFORCE · Batch baseline and group–batch comparison · Back to discussion

Q05 · DISCUSSION BACKUP

What if every trajectory gets the same reward?

R₁ = ⋯ = R_B　⟹　A₁ = ⋯ = A_B = 0

Consequence

The batch has zero reward-driven policy gradient.

Possible responses

Difficulty curricula, mixed tasks, exploration or resampling.

For independent equal-success-probability samples: P(uniform batch) = pᴮ + (1 − p)ᴮ.

Token reweighting cannot create a missing success signal.

Direct consequence of batch centering; responses are proposed extensions · Back to discussion

Q06 · DISCUSSION BACKUP

How should noisy rewards be handled?

Policy mismatch

Monitor ratios and drift.
Control update size and queue delay.

Reward uncertainty

Calibrate scores and inspect outliers.
Consider repeated scoring or conditional baselines.

01
A gate based on policy probabilities does not test reward reliability.
02
More independent samples can reduce noise, while systematic reward bias remains.

Dedicated noisy-reward experiments are still needed.

NetEase discussion notes · Q6 · Back to discussion

Q07 · DISCUSSION BACKUP

Can averaging dilute long-trajectory learning?

Reduction	Explicit trajectory weight
Sample mean	1 / B
Token mean	Tᵢ / ∑ⱼ Tⱼ

01
Tᵢ counts policy tokens, not the number of interaction turns.
02
Averaging removes extra length weight but can dilute a few decisive token updates.
03
The tool-use ablation supports this tradeoff in one setting, not universally.

Main talk: Sample mean and loss-reduction continuation · Back to discussion

Q08 · DISCUSSION BACKUP

What helps when successful rewards are sparse?

Optional failure-token filter	Peak mean (%)	Peak step
No filtering, q = 1.0	37.0	600
Light filtering, q = 0.9	39.4	700
Aggressive filtering, q = 0.2	32.2	200

7B Python tools. Each row reports its own peak, not a matched checkpoint.

Filtering changes both token selection and total negative-trajectory weight.

FlashREINFORCE · Appendix, optional negative-token filtering · Back to discussion

Q09 · DISCUSSION BACKUP

How should lag and δ be interpreted?

01
Version lag and measured policy drift have no fixed conversion.
02
Passing a lower-bound KL proxy can still miss a large full-distribution shift.
03
Negative advantage reduces sampled-action probabilities; it does not directly optimize early stopping.

Observed token-mean continuation: length 2,573 to 3,947; truncation 18.8% to 45.3%.

The length-collapse mechanism needs EOS and token-gradient diagnostics.

Main talk: Sequence screen and loss-reduction ablation · Back to discussion

Q10 · DISCUSSION BACKUP

Can rejected rollouts erase the efficiency gain?

01
Measure token and time costsA small number of rejected long trajectories may consume substantial compute.
02
Stratify the rejection rateInspect trajectory length, difficulty, lag and training stage.
03
Compare time to a target qualityInclude generation, queuing, rejection and learning efficiency.

The reported 0.13% Math trajectory rejection rate does not generalize to all agent workloads.

FlashREINFORCE · Threshold discussion; proposed efficiency diagnostics · Back to discussion

Q11 · DISCUSSION BACKUP

Which memory savings introduce approximation?

Design choice	What changes
No critic	Baseline estimation
Token-only IS	History mismatch remains
Sampled-action KL	Admission statistic and behavior storage
Sample mean	Trajectory-length weighting

No systematic memory–gradient-error–convergence frontier is reported.

Compare fixed-batch gradient error, then final quality, memory and time.

FlashREINFORCE · Local surrogate and practical method · Back to discussion

Q12 · DISCUSSION BACKUP

What training diagnostics are most useful?

01
Reward and task difficultyCheck calibration and whether the sampled tasks produce learning signal.
02
Behavior probabilitiesUse the distribution that actually generated the sampled tokens.
03
Joint training dynamicsTrack reward, entropy, length, truncation, tools, ratios and drift.
04
Controlled comparisonsChange one component and report both quality and compute cost.

NetEase discussion notes · Training practice · Back to discussion

FlashREINFORCE

Learning from each completed interaction

Agent trajectories finish at different times

Group-relative learning

Asynchronous learning

Rollout budget also determines coverage

32 prompts × 4 rollouts

128 prompts × 1 rollout

What changes with a single rollout?

An asynchronous, one-pass learner

Policy snapshots μᵢ

Next B completions

One update

A local off-policy surrogate

Behavior quantities stay fixed

The action distribution changes

What token importance sampling corrects

Exact conditional correction

Residual history mismatch

Surrogate error accumulates with drift

FlashREINFORCE

01 · One-batch REINFORCE

02 · Sequence trust

03 · Sample mean

Signed feedback without sibling rollouts

A low-cost sequence drift screen

Trajectory length changes update weight

Why normalize?

What remains unresolved?

The complete update

One fresh batch, one optimization step

Reasoning, tool use, and interaction

Higher mean accuracy with half the rollouts

Stable training through 6,000 updates

Length decreases as entropy settles

Tool use persists during training

Scaling to Qwen3-30B-A3B at lag 8

Sustained performance at policy lag 8

ALFWorld seen and unseen success

Fresh updates and signed feedback matter

Freshness at equal budget

Signed reward feedback

Sample mean avoids the observed length collapse

The granularity of admission matters

Token-local masking

Sequence admission

Group and batch centering remain close

128 prompts × 1

32 prompts × 4

What the evidence does not establish

A practical recipe for asynchronous agent RL

Paper, code, and related work

Questions & discussion

How much does sequence trust contribute?

Is the gate necessary or a safeguard?

Does centering extend to continuous rewards?

Continuous terminal score

Dense stepwise rewards

Does a hard success get negative advantage?

What if every trajectory gets the same reward?

Consequence

Possible responses

How should noisy rewards be handled?

Policy mismatch

Reward uncertainty

Can averaging dilute long-trajectory learning?

What helps when successful rewards are sparse?

How should lag and δ be interpreted?

Can rejected rollouts erase the efficiency gain?

Which memory savings introduce approximation?

What training diagnostics are most useful?

Slide navigatorClick a slide to jump · Esc to close

Keyboard & controls

Flash
REINFORCE