Wen's Blog

My thoughts on deep learning and AI

From Weight Decay to Hyperball Optimization (Part 2): Weight decay theory deep dive

Part 2 of a two-part series: why weight decay sets effective step size in scale-invariant nets.

26 min read · December 14, 2025

2025 · weight decay hyperball optimization deep learning · research
From Weight Decay to Hyperball Optimization (Part 1): Hyperball optimizer + intuition

Part 1 of a two-part series: Hyperball optimization and empirical intuition.

12 min read · December 14, 2025

2025 · weight decay hyperball optimization deep learning · research