Scaling DoRA: High-Rank Adaptation via Factored Norms and Fused Kernels

Researchers have made a significant breakthrough in improving the efficiency of Weight-Decomposed Low-Rank Adaptation (DoRA) by introducing factored norms and fused kernels. The original DoRA method required substantial computational resources, particularly for large input dimensions, due to the need to materialize the dense product of two matrices. By decoupling weight magnitude from direction, the new approach enables high-rank adaptation while reducing memory requirements. For instance, at input dimension 8192 and rank 384, the modified method significantly decreases the transient working memory needed for a single module's norm calculation from approximately 512 MB in bf16. This advancement has significant implications for large-scale machine learning applications, especially in scenarios where memory constraints are a limiting factor¹. The ability to efficiently adapt models to new tasks and environments is crucial for practitioners, as it enables more effective deployment of AI systems in real-world settings, where adaptability and scalability are essential.