Kaiyue Wen
Ph.D. Student at Stanford University
Stanford University
Stanford, CA
I am a Ph.D. student at Stanford University, advised by Tengyu Ma and Percy Liang. Previously, I studied at Tsinghua University’s Yao Class. I study the science of deep learning. My work combines theory and experiments to understand how deep learning works and develop new algorithms. Recently, I have become interested in using AI to accelerate understanding of deep learning.
Optimization and Training Dynamics. How neural networks learn depends fundamentally on their optimization dynamics. My work examines optimizer scaling, Hyperball, and the river valley structure of pretraining loss.
Reasoning and Self-Improvement. I also study how to elicit and measure language model capabilities from first principles. My work explores critic-guided reinforcement learning and measures idea generation capabilities through the information needed to recover future research ideas.
During my previous studies, I was fortunate to be advised by Zhiyuan Liu, Andrej Risteski, Jingzhao Zhang, Yuhao Wang, and Zhiyuan Li.
News
| Apr 02, 2026 | Gave a talk at Kimi on Hyperball. |
|---|---|
| Mar 21, 2026 | Gave a talk at the IOS INFORMS Optimization Society Conference on recent works around Fantastic Optimizers & Hyperball! [slides] |
| Jan 26, 2026 | 2 papers accepted at ICLR 2026! |
| Nov 26, 2025 | Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free received a NeurIPS 2025 Best Paper Award! [announcement] |
| Sep 01, 2025 | New preprint (Fantastic Pretraining Optimizers and Where to Find Them) on arxiv! |
Selected Research
Optimization and Training Dynamics
ICLR 2026
Fantastic Pretraining Optimizers and Where to Find Them
Rigorous optimizer benchmarking shows that vanilla Muon's efficiency advantage over AdamW shrinks with overtraining under constant weight decay.
2026 preprint
Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization
Hyperball uses weight-normalized training to improve Muon's efficiency.
ICLR 2025
Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective
Pretraining loss exhibits a river valley landscape, which resembles a deep valley with a river at its bottom. This explains how training with a high learning rate can make more progress than training with a low learning rate, even when its intermediate losses are higher.
Reasoning and Self-Improvement
2026 preprint
Trust the Critic More
A learned critic can guide reasoning RL using partial solutions. On Qwen3-4B trained on FineProofs-RL, AC2 exceeds GRPO's best mean score on IMO-ProofBench with approximately 60% fewer decoding FLOPs.
2026 preprint
Priced Guidance: Can Language Models Generate Future Research Ideas?
Priced Guidance uses “20 Questions” to measure the amount of information an LLM needs to recover a future research idea. This yields a lower bound on the probability of generating that idea without hints.