SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation

Researchers have introduced SkMTEB, a benchmark for evaluating text embedding models in the Slovak language, which has limited digital resources. This benchmark comprises 31 datasets across seven task types, significantly expanding the coverage of existing multilingual benchmarks for Slovak. An evaluation of 31 embedding models revealed that large, instruction-tuned multilingual models achieve the strongest performance. The development of SkMTEB has implications for natural language processing in low-resource languages, as it provides a comprehensive framework for assessing the effectiveness of text embedding models¹. The creation of such benchmarks is crucial for improving the accuracy and reliability of language models, particularly in languages with limited digital presence. This matters to practitioners because it enables the development of more effective language models for low-resource languages, which can have significant implications for various applications, including cybersecurity and threat detection.