Princeton University · Research Talk

Reasoning
Machines

Principled architectures and learning algorithms for large language models.

Yifan Zhang Princeton University · 2026 · ~30 min

Press → or Space to begin · ? for help

Research agenda

Three pillars

I · Architectures Efficient & expressive sequence models Low-rank, structured attention state. TPA · GRAPE · HLA · Fast-Weight Attention

II · Learning Scalable RL algorithms Objectives derived, not tuned. RPG · SDPG · General Preference Model

III · Agents Reasoning & agentic systems From reasoning to acting. Proposer-Verifier · Meta Prompting · MathCode · Web World Models

NeurIPS Spotlight · methods adopted by Thinking Machines & DeepSeek

Selected Works

[SDPG] Self-Distilled Policy Gradient

Yifeng Liu*, Shiyuan Zhang*, Yifan Zhang*, Quanquan Gu

arXiv:2606.04036

[Falcon] Fast-Weight Attention for Continual Learning

Yifan Zhang et al.

Preprint

[FlashSampling] Fast & Memory-Efficient Exact Sampling

Tomas Ruiz*, Zhen Qin*, Yifan Zhang†, et al.

arXiv:2603.15854

[DDL] Deep Delta Learning

Yifan Zhang, Yifeng Liu, Mengdi Wang, Quanquan Gu

arXiv:2601.00417

[GRAPE] Group Representational Position Encoding

Yifan Zhang, Zixiang Chen, Yifeng Liu, et al.

ICLR 2026

[RPG] KL-Regularized Policy Gradient for LLM Reasoning

Yifan Zhang*, Yifeng Liu*, Huizhuo Yuan, et al.

ICLR 2026 · Tinker · DeepSeek-V3.2

[TPA] Tensor Product Attention Is All You Need

Yifan Zhang*, Yifeng Liu*, Huizhuo Yuan, et al.

NeurIPS 2025 Spotlight

[GPM] Beyond Bradley–Terry: General Preference Model

Yifan Zhang*, Ge Zhang*, Yue Wu*, Kangping Xu, Quanquan Gu

ICML 2025

[Proposer–Verifier] Cumulative Reasoning

Yifan Zhang*, Jingqin Yang*, Yang Yuan, Andrew C Yao

TMLR

* equal contribution · † corresponding author

Impact

Research that ships

1,387

Citations (Google Scholar)

19

Peer-reviewed papers (NeurIPS · ICLR · ICML · ACL · AAAI · TMLR)

15

Peer-reviewed first-author papers (incl. co-first)

2

Spotlights: TPA (NeurIPS 2025) & Beyond Squared Error (ICLR 2025)

2

Frontier labs adopting RPG — Thinking Machines & DeepSeek

2.8k+

GitHub stars on open-source projects

✦ Most-cited636 Cumulative Reasoning · 99 Meta Prompting · 97 Iterative Question Composing · 51 AutoMathText · 47 Contrastive Learning Theory · 46 Tensor Product Attention · 40 Diagram of Thought · 37 General Preference Model

★ Most-starred574 MathCode · 458 TPA · 355 Deep Delta Learning · 308 Cumulative Reasoning · 302 Meta Prompting · 101 HLA · 95 GRAPE · 91 AutoMathText

The question

Can we build machines that reason — efficiently, reliably, and at scale?

01Today's models are memory-bound — attention state grows with every token.
02Their learning is brittle and heuristic — a patchwork of RL tricks.
03Their reasoning is unstructured and opaque — longer chains, little theory.

My research makes each of these principled — and ships the result.

Pillar I

Architectures

Structure the state — expressive and efficient.

The problem

Attention is memory-bound

01The KV cache grows linearly with context — the wall for long-context and inference.
02Compute is quadratic in sequence length.
03Today's fixes (MQA / GQA) trade quality for memory.

What if the state itself were low-rank?

NeurIPS 2025 · Spotlight

Tensor Product Attention

Factorize queries, keys, and values as contextual tensor products — compressing the KV cache by up to ~10×.

$\;Q_t=\sum_{r=1}^{R}\,a^{Q}_{t,r}\otimes b^{Q}_{t,r}\,,\qquad$ likewise for $K_t,\,V_t$ — rank $R \ll d$.

→Longer context at fixed memory; unifies MHA / MQA / GQA as special cases.
→Fully RoPE-compatible; the T6 backbone trains better at equal budget.

"Tensor Product Attention Is All You Need."

A family of methods

One principle, many forms

GRAPE Group-representational position encoding — symmetry-aware positions. ICLR 2026

Higher-order Linear Attention Linear-time state with higher-order token interactions. Preprint · HLA

Fast-Weight Attention Fast weights as memory for continual learning. Preprint · Falcon

Deep Delta Learning The delta rule, made deep — error-driven state updates. Preprint · DDL

Throughline: structured state → linear cost, lasting memory.

Pillar II

Learning

Derive the objective — don't tune the heuristic.

The problem

Policy gradients for LLMs are a zoo

01RLHF and RLVR now drive frontier reasoning.
02But GRPO / PPO are a patchwork of clipping and KL hacks — unstable, hard to reason about.

Start from the KL-regularized objective — and be exact.

ICLR 2026

KL-regularized policy gradient, done right

A unified design space for KL-regularized policy-gradient algorithms — with the correct gradient and stop-gradient treatment.

Impact Adopted in Thinking Machines' Tinker and DeepSeek-V3.2.

→More stable training; recovers and repairs popular methods as special cases.

Alignment & RL

Beyond rewards and clips

Self-Distilled Policy Gradient The policy distills into its own target — stable, low-variance updates.

$\nabla_\theta\,\mathbb{E}_{a\sim p_t}\!\big[-\log p_t(a)\,\mathrm{SG}(\log \bar q_t/\bar p_t+\bar D_t)\big]$

SDPG

General Preference Model Beyond Bradley–Terry: a skew-symmetric operator captures cyclic, intransitive human preference. ICML 2025

Reward modeling and RL, on one principled footing.

Pillar III

Agents

From reasoning to systems that act.

Reasoning

Reasoning as structure, not length

Structured inference Cumulative Reasoning & Diagram of Thought — reasoning as a DAG, not a linear chain. Proposer–verifier loops that build and check intermediate results.

Control Meta Prompting — structured prompting as a control layer for LLMs. Composable, reusable reasoning scaffolds for AI systems.

Better structure beats longer chains.

Agentic systems

Agents that act on the world

Flagship MathCode — a terminal coding agent that formalizes natural-language math into Lean 4 and proves it.

→Lanser-CLI — RL from compiler & language-server feedback (cf. Claude Code v2.0.74).
→Web World Models — controllable, open-ended worlds for language agents: rules in web code, context from LLMs.

Agents that browse, compute, and prove.

What's next

Toward reasoning machines

01Million-token memory — structured, fast-weight state at scale.
02Verifiable RL — learning from compilers, language servers, and world models (Lanser-CLI; Web World Models).
03Agentic reasoning — structured, tool-using agents with principled objectives.

Principled methods, built to ship.

Thank you

Thank you.

With gratitude to my collaborators and advisors.

[email protected] yfz.ai Google Scholar GitHub

Reasoning
Machines

Three pillars

Selected Works

Research that ships

Can we build machines that reason — efficiently, reliably, and at scale?

Architectures

Attention is memory-bound

Tensor Product Attention

One principle, many forms

Learning

Policy gradients for LLMs are a zoo

KL-regularized policy gradient, done right

Beyond rewards and clips

Agents

Reasoning as structure, not length

Agents that act on the world

Toward reasoning machines

Thank you.

Slide navigatorClick a slide to jump · Esc to close

Keyboard & controls

ReasoningMachines

Three pillars

Selected Works

Research that ships

Can we build machines that reason — efficiently, reliably, and at scale?

Architectures

Attention is memory-bound

Tensor Product Attention

One principle, many forms

Learning

Policy gradients for LLMs are a zoo

KL-regularized policy gradient, done right

Beyond rewards and clips

Agents

Reasoning as structure, not length

Agents that act on the world

Toward reasoning machines

Thank you.

Slide navigatorClick a slide to jump · Esc to close

Keyboard & controls

Reasoning
Machines