Do VLMs Need Vision Transformers? Evaluating State Space Models as Vision Encoders

Researchers have been investigating whether state space models can serve as a viable alternative to traditional transformer-based encoders in large vision-language models. By systematically evaluating state space model vision backbones in a controlled setting, they aim to determine their effectiveness. The standard approach typically employs a frozen vision backbone, which is then connected to a large language model through a lightweight connector. However, the use of state space models as vision encoders may offer a different paradigm. This line of inquiry is particularly relevant given the rapid advancements in large language models, which have significant implications for both capability and risk surfaces¹. The security implications of these developments are still being explored, and understanding the role of state space models in vision-language models can help mitigate potential risks. So what matters to practitioners is that alternative vision encoders like state space models can potentially offer new avenues for securing vision-language models.