Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?

The effectiveness of AdamW, a widely used optimizer for training large language models, is being questioned under heavy-tailed noise conditions. Despite its widespread adoption, the theoretical foundations of AdamW are largely based on finite-variance regimes, which may not accurately reflect real-world scenarios. Recent studies have demonstrated that sign-based optimizers, such as Lion and Muon, can achieve superior performance in heavy-tailed environments, casting doubt on AdamW's efficacy¹. The implications of this are significant, as large language models are increasingly being used in critical applications. If AdamW is indeed suboptimal under heavy-tailed noise, it could have far-reaching consequences for the development and deployment of AI systems. This raises important questions about the reliability and robustness of these models, and highlights the need for further research into optimization techniques that can handle complex, real-world noise distributions. So what matters to practitioners is that the choice of optimizer can have a profound impact on the performance and reliability of their AI models.