Princeton University · Research Talk
Auto
Research
Toward autonomous superintelligence: principled architectures and agentic RL that automate AI research.
Press → or Space to begin · ? for help
Research agenda
Toward autonomous superintelligence
The route is AutoResearch: automating AI research itself, so improvements compound. That goal sets three pillars.
NeurIPS Spotlight · methods adopted by Frontier Labs
The thesis
Superintelligence as a loop, not a leap
- 01The goal is AutoResearch: automating AI research itself, so progress compounds.
- 02Architectures and agentic RL are not separate interests; they are what the loop demands.
Selected works · 2026
Recent work
* Equal contribution · † Corresponding author
Selected works
Architectures and learning algorithms
* Equal contribution · † Corresponding author
Impact · citation / star counts: September 2026 snapshot
Research that ships
✦ Most-cited704 Cumulative Reasoning · 99 Meta Prompting · 97 Iterative Question Composing · 51 AutoMathText · 47 Contrastive Learning Theory · 46 Tensor Product Attention · 40 Diagram of Thought · 37 General Preference Model
★ Most-starred732 MathCode · 460 TPA · 355 Deep Delta Learning · 308 Cumulative Reasoning · 317 Meta Prompting · 103 HLA · 115 GRAPE · 92 AutoMathText
◆ In productionAutoMathText Kimi K2 · K3 · NVIDIA Nemotron 3 | GPO Kimi K1.5 · K2 · K3 RL | GRAPE-A OpenAI · SpaceXAI · Thinking Machines · Jane Street | RPG Thinking Machines · DeepSeek
The question
What would it take to automate AI research itself?
- 01Models that hold long horizons: today's attention is memory-bound, and learning stops after pretraining.
- 02RL that scales: today's policy gradients are a brittle patchwork of heuristics.
- 03A loop that closes: model and harness must coevolve for improvements to compound.
My research makes each piece principled, and ships the result.
Pillar I
Architectures
Structure the state — expressive and efficient.
The problem
Attention is memory-bound
- 01The KV cache grows linearly with context — the wall for long-context and inference.
- 02Compute is quadratic in sequence length.
- 03Today's fixes (MQA / GQA) trade quality for memory.
What if the state itself were low-rank?
Model–hardware co-design
Model structure sets the workload
| Component | Cost it controls | Research direction |
|---|---|---|
| Attention | KV memory and context reads | TPA · FlashTPA |
| Recurrent state | Memory carried across tokens | Falcon · HLA |
| Depth and residuals | Parameter reuse and stable updates | DeepLoop · RLT · DDL |
| Position | Relative geometry and cache reuse | GRAPE |
| LM head | Logits traffic and sampling kernels | FlashSampling |
Architecture determines the state and operations; kernels determine how efficiently the hardware executes them.
Research map · TPA ↗ · FlashSampling ↗
NeurIPS 2025 · Spotlight
Tensor Product Attention
Factorize queries, keys, and values as contextual tensor products — compressing the KV cache by up to ~10×.
- →Longer context at fixed memory; unifies MHA / MQA / GQA as special cases.
- →Compatible with any positional encoding (RoPE, ALiBi, additive RPE, GRAPE); the T6 backbone trains better at equal budget.
"Tensor Product Attention Is All You Need."
Pillar I · Tensor Product Attention
The KV cache is the long-context wall
- →Queries, keys, and values factor as contextual tensor products of rank R ≪ d.
- →Unifies MHA / MQA / GQA as special cases; compatible with any positional encoding, e.g., RoPE, ALiBi, additive RPE, GRAPE.
NeurIPS 2025 Spotlight · 460 GitHub stars
Pillar I · TPA · FlashTPA Decoding
FlashTPA: decode on the factors
- →Per-token attention FLOPs scale as $\Theta(M(R_Q R_K D + H R_Q R_K + H R_V E))$: with small ranks TPA saves prefill and decode compute, not just KV-cache memory.
- →Benchmarked against FlashMHA, FlashGQA, FlashMQA, and FlashMLA: faster decoding at long sequence lengths (up to 512K tokens), with Triton decode and prefill kernels in the repo.
Algorithms 2 & 3 in the paper · arXiv:2501.06425 · CUDA kernel in progress
Pillar I · FlashSampling · arXiv:2603.15854
FlashSampling: sample without the logits
- →Exact, not approximate: tilewise Gumbel-max keeps one maximizer per row and per vocabulary tile; a small reduction over tiles finishes the categorical sample.
- →Fused into the LM-head matmul: avoids logits materialization in HBM. Kernels reach 2.23× over compiled PyTorch multinomial; vLLM TPOT falls up to 10.2% on Qwen3-1.7B (B200).
Ruiz*, Qin*, Zhang†, Shen, Zhong, Wang† · revised May 2026 ↗
Pillar I · architecture families
Structure in position, state, and depth
| Method | Structured object | Consequence |
|---|---|---|
| GRAPE | Group action on positions | Exact relative-position law |
| HLA | Prefix sufficient statistics | Higher-order mixing in linear time |
| Falcon | Fast-weight write rule | Online learning in a bounded state |
| DDL | Residual transition operator | Learned erasure and reflection |
| DeepLoop | Repeated parameter visits | Loop-aware residual scaling |
| RLT | Final-state feedback across tokens | Computation paths grow with time |
Each mechanism controls a different part of the computation.
Pillar I · GRAPE · ICLR 2026
Positions as group representations
One frame, exact relative position laws, streaming cacheability · see also OpenAI's GPT-5, xAI's Grok 4, Thinking Machines' Inkling, and Jane Street's blog
Pillar I · DeepLoop · the framework
Loop the blocks, scale the depth
- →A looped Transformer stores K physical blocks once and revisits them for R rounds: effective depth N = KR with no new parameters.
- →Because every visit reuses the same weights, residual updates align across rounds; that alignment is exactly what the α, β rule accounts for.
Pillar I · DeepLoop · arXiv:2607.13491
Loop-aware depth scaling that shows up in loss
Gap widens with loop count; recovers DeepNorm's 1/4 when visits decorrelate; also lifts HRM voted accuracy on ARC-AGI.
Pillar I · Recurrent Looped Transformer
Carry computation across tokens
Encoder: known tokens build causal global KV memory in parallel.
Decoder: the previous final state enters the next token’s update; each layer also keeps its own SWA cache.
RLT · October 5, 2026 ↗ · Full RLT talk ↗
Hidden-state feedback continues across prompt and response. Global encoder memory still grows with context.
Pillar I · RLT · three-seed results
Feedback supports length generalization
RLT · October 5, 2026 ↗ · Mean ± sample SD, n = 3; best in-distribution checkpoint.
Training lengths: parity ≤40, swaps ≤32. Equal layer counts; parameter counts and compute differ.
Pillar I · RLT · execution tradeoff
Feedback frequency controls parallelism
| Decoder schedule at 4+4 | Feedback | Parity / 64 | S₅ swaps / 64 |
|---|---|---|---|
| RLT-1 | Every token | 100.00% | 100.00% |
| RLT-2 | Every 4 tokens | 98.99% | 19.60% |
| RLT-0 | None | 50.23% | Near chance |
- 01Known tokens in a chunk run together. Four-token chunks give 2.27× faster CPU training steps on the mod-5 tasks at 4+4.
- 02State tracking depends on the feedback interval. Chunking preserves parity much better than permutation tracking. Generated tokens still arrive sequentially.
RLT · October 5, 2026 ↗ · Accuracy: three-seed means. Timing: seed 42, four CPU threads, FP32.
Replay rebuilds current-policy state and caches; the recorded sampler probabilities remain fixed.
Pillar I · fixed-state models
A state that stays fixed, and keeps learning
- →HLA: higher-order token interactions at linear time; the state does more per byte.
- →Falcon: fast weights as episodic memory, so learning continues after pretraining ends.
Pillar I · Higher-order Linear Attention · Zhang, Qin, Gu
HLA: higher-order attention, linear time
- →Higher-order interactions from compact prefix sufficient statistics: closed-form streaming identities and a strictly causal masked variant with two extra summaries.
- →Chunk-parallel training via associative scans reproduces the serial recurrence exactly; extends to third order and higher.
Preprint · 103 GitHub stars · attention-like data-dependent mixing at recurrent-model cost
Pillar I · HLA · the second-order mechanism
Two summaries are enough
- →Second-order tensor attention $Q(K^{\top}K)Q^{\top}$ touches the context only through second moments, so two streaming summaries are sufficient statistics.
- →An associative semidirect-product operator turns the recurrence into a parallel scan; unnormalized and normalized variants share the same state.
Strict causality via a masked variant with two extra summaries · extends to third order and higher
Pillar I · Falcon · ByteDance Seed Technical Report; arXiv:2608.27763
Falcon: fast weights as continual learning
- →One lens: attention, recurrent fast-weight memories, and selective SSMs are write rules over a bounded memory.
- →Compressing a growing context into a bounded state makes the write rule an online continual-learning rule, connecting architecture design to learning theory.
The continual-learning perspective behind the fixed-state family
Pillar I · Falcon · six write rules
The Falcon family
- →Read-after-write alignment: the prefix-aligned pair $(\phi(k_{t-1}), v_t)$ trains the fast memory on what was actually available at prediction time.
- →Consistent semantics: $\beta_t$ plasticity gain, $\lambda_t$ ridge shrinkage, $\eta_t$ induced step size; smoothness-matched or energy-normalized.
Aligned, normalized updates keep language-model quality and improve arithmetic length extrapolation
Pillar I · Falcon · one gradient step per token
The write is a gradient step
- →The write is literally one gradient-descent step on an instantaneous ridge objective; the residual $r_t=y_t-S_{t-1}^{\top}x_t$ is measured against the previous state's prediction, unlike standard DeltaNet.
- →The loss is $L_t$-smooth with $L_t=\|x_t\|_2^2+\lambda_t$, so $\eta_t$ is smoothness-matched; with $\lambda_t=\varepsilon=0$ it reduces exactly to classical NLMS.
Pillar I · Falcon · experiments
Aligned writes extrapolate
- →Language modeling holds: 124M–130M models on FineWeb-Edu at a matched ~50B-token budget stay competitive with Transformer, Mamba-2, DeltaNet, and Gated DeltaNet baselines.
- →Length extrapolation improves: the shifted, normalized writes win where storage and carry propagation dominate, 87.2 vs 65.8 mean accuracy out of distribution.
A controlled diagnostic that isolates the memory-writing behavior of the recurrent state
Pillar I · Deep Delta Learning · arXiv:2601.00417
DDL: gating the residual spectrum
- →A rank-1 Householder-style update on the identity shortcut: erasing along k and writing v are coupled, with the gate as a synchronous dynamic step size.
- →The network controls the spectrum of its layer transition operator, modeling non-monotonic dynamics while keeping gated-residual training stability.
arXiv:2601.00417 · 355 GitHub stars · Zhang, Liu, Wang, Gu
Pillar II
Learning
Derive the objective, then scale agentic RL to long horizons.
Pillar II · a unifying view
REINFORCE is all you need
- →Pretraining is off-policy REINFORCE with A ≡ 1: the corpus is the behavior policy, and every next token is a positively rewarded action.
- →Scaling REINFORCE across the spectrum, off-policy (corpus, replayed rollouts, IS-corrected) to on-policy (verifier-rewarded rollouts), unifies the training stack.
One objective family from pretraining to agentic RL
Pillar II · FlashREINFORCE
Single-rollout asynchronous REINFORCE
FlashREINFORCE · NVIDIA Technical Report ↗ · Full FlashREINFORCE talk ↗
mᵢ admits a complete trajectory. Token ratios correct actions at stored histories, not the history distribution.
Pillar II · FlashREINFORCE · experiments
Reasoning, tools, and long-horizon agents
| Setting | Result | Comparison / protocol |
|---|---|---|
| Long-CoT · R1-Distill 1.5B | 21.7 → 33.7 | AIME24/25 mean; 6,000 updates, lag ≈4 |
| Qwen2.5-Math-1.5B | 38.0 vs 36.3 | Five-benchmark mean; 256k vs 512k rollouts |
| Python tools · Qwen3-30B-A3B | +6.8 points | Matched-budget GRPO comparison |
| ALFWorld · Qwen2.5-7B | 98.3% / 96.5% | Seen / unseen; 12.8k training trajectories |
One rollout per prompt removes sibling-response synchronization and broadens prompt coverage.
FlashREINFORCE · NVIDIA Technical Report ↗ · Results from the public technical report; benchmark means are percentages.
The problem
Policy gradients for LLMs are a zoo
- 01RLHF and RLVR now drive frontier reasoning.
- 02But GRPO / PPO are a patchwork of clipping and KL hacks — unstable, hard to reason about.
Start from the KL-regularized objective — and be exact.
ICLR 2026
KL-regularized policy gradient, done right
A unified design space for KL-regularized policy-gradient algorithms — with the correct gradient and stop-gradient treatment.
- →More stable training; recovers and repairs popular methods as special cases.
Pillar II · RPG · ICLR 2026
One design space, every KL policy gradient
- →Every cell gets the correct gradient and stop-gradient treatment, derived from the objective.
- →Recovers and repairs GRPO-style methods as special cases; measurably more stable training.
Adopted in Thinking Machines' Tinker and DeepSeek-V3.2
Pillar II · KL-Regularized Policy Optimization
KLPO: anchor the update at the sampler
Fit the Gibbs improvement step under the actual behavior policy q.
- 01Regress the optimality condition. Profile out the intercept instead of learning a prompt-dependent normalizer.
- 02One complete rollout per prompt. Terminal returns support critic-free updates, including stochastic tool outputs.
- 03No multiplicative importance weights. Sampler probabilities enter through log-ratios.
KLPO · October 5, 2026 ↗ · Yifan Zhang · On KL-Regularized Policy Optimization
S is the improvement signal; β is the KL coefficient; Z normalizes the conditional policy.
Pillar II · KLPO · October 2026 default
Token regression with Monte Carlo KL
$\widehat g=-\sum_u h_u\!\left[\nabla\log p(a_u)-\dfrac{1}{M}\sum_{j=1}^{M}\nabla\log p(v_{u,j})\right]$
- 01M ≥ 1 auxiliary tokens per prefix. Draw from q independently of the complete rollout; no extra completions.
- 02Center scores under the sampler. Independent MC-KL recovers the Full-KL token gradient in expectation.
- 03Fresh draws preserve the stated unbiasedness. Reusing fixed historical-sampler records defines an empirical surrogate.
KLPO · October 5, 2026 ↗ · Formula: one-response loss gradient; probabilities condition on the stored prefix.
Released: theory, CPU-verified losses and Molt integration. Paper-scale GPU results are not reported.
Alignment · ICML 2025
General Preference Model
Beyond Bradley–Terry: a skew-symmetric operator captures cyclic, intransitive human preference.
Beyond Bradley–Terry Models: A General Preference Model for Language Model Alignment ↗
Agentic RL at scale
The framework and the science
Scaling agentic RL takes both the infrastructure and the science.
Pillar II · Molt · NVIDIA Technical Report; arXiv:2607.21653
Molt: three boxes, one async loop
- →About 9.2K lines of RL code; MoE-native to 1T-class, think DeepSeek-V3 at EP 256.
- →reinforce · rloo · grpo · gae · on-policy distillation; router replay (R3) and IS correction for async rollout.
Pillar II · RL Kernel Mismatch · 08/2026
Same checkpoint, different policy
- →Chunkwise, scan, and recurrent kernels are algebraically equal but different finite-precision operators; DeltaNet's corrective write feeds the error into future updates.
- →Four remedies: actor-emitted importance ratios, one canonical execution rule, higher-precision state, and exact modified rejection sampling over a fast proposal.
Pillar III
AutoResearch
Agents that automate research itself.
Reasoning
Reasoning as structure, not length
Better structure beats longer chains.
Pillar III · structured reasoning
A chain forgets; a DAG accumulates
- →A proposer suggests steps; a verifier checks them; only verified results join the context.
- →Cumulative Reasoning (TMLR, 704 citations) and Diagram of Thought formalize the DAG view.
Agentic systems
Agents that act on the world
- →Lanser-CLI — RL from compiler & language-server feedback (cf. Claude Code v2.0.74).
- →Web World Models — controllable, open-ended worlds for language agents: rules in web code, context from LLMs.
Agents that browse, compute, and prove.
Pillar III · MathCode
From natural language to verified proof
- →A terminal coding agent where the compiler is the referee; 732 GitHub stars, the most-starred project.
- →CriticLean (ACL 2026) adds critic-guided RL to the formalization step itself.
Pillar III · environments
Rewards with executable ground truth
Executable checks make success reproducible; the specification and evaluator still need to match the intended task.
Pillar III · closing the loop
Model and harness coevolve
- →We build our own harness; token-exact traces survive even opaque context compaction.
- →This loop is what turns research agents into AutoResearch.
Pillar III · Agora · collective AutoResearch
Git records the research process
- 01Append-only contributions. Code, hypotheses, results and verifications form a dependency graph that workers can check out and reproduce.
- 02Evidence guides attention. Reuse and verification propagate scores; analyze() exposes frontier results, contested claims and underexplored branches.
Agora · arXiv:2609.18094 ↗ · NeurIPS 2026 AutoMLR Workshop Oral · NVIDIA Technical Report
Workers choose experiments from shared state; there is no central planner assigning tasks.
Pillar III · Agora · sustained research run
Collective search leaves reproducible evidence
141 donors, 32 architecture families. Transfer into a 119.6M-parameter attention–SSM hybrid using donor weights and forward passes.
3.3923 → 1.899 bpb. The development evaluator scores 200 FineWeb-Edu texts; pretraining and fine-tuning are prohibited.
Agora · arXiv:2609.18094 ↗ · One weight-transfer study; bpb is the task’s development-evaluator score.
What's next
Toward autonomous superintelligence
- 01AutoResearch: automate AI research itself, so improvements compound instead of being hand-built.
- 02Million-token memory: architectures for long context and continual learning, fast-weight state at scale.
- 03Agentic RL scaling: single-rollout learning with FlashREINFORCE and sampler-anchored KLPO.
- 04Collective AutoResearch: use Agora to share experiments, verify results, and allocate the next research effort.
Principled methods, built to ship.
Thank you
Thank you.
With gratitude to my collaborators and advisors.