C$^{2}$R: Cross-sample Consistency Regularization Mitigates Feature Splitting and Absorption in Sparse Autoencoders

Researchers have identified significant challenges in Sparse Autoencoders (SAEs) when applied to large language models, including feature splitting and absorption. Feature splitting occurs when coherent concepts are fragmented into non-atomic latents, while feature absorption creates arbitrary exceptions in generated text. To mitigate these issues, a new technique called Cross-sample Consistency Regularization (C$^{2}$R) has been proposed¹. This method regularizes SAEs to promote consistency across different input samples, reducing the occurrence of feature splitting and absorption. By addressing these challenges, C$^{2}$R enables more effective decomposition of activations into sparse, human-understandable features. This advancement has significant implications for the interpretation of large language models, allowing for more accurate and reliable analysis. So what matters to practitioners is that C$^{2}$R can improve the robustness and reliability of SAEs, enabling more trustworthy insights into complex language models.