World Properties without World Models: Recovering Spatial and Temporal Structure from Co-occurrence Statistics in Static Word Embeddings

Research reveals that spatial and temporal structure can be recovered from static word embeddings, such as GloVe and Word2Vec, using ridge regression probes¹. This challenges the notion that large language models are required to capture world-like internal representations. By analyzing co-occurrence statistics in these embeddings, researchers found that much of the relevant structure is already latent in the text itself. The study applied the same class of probes used to interpret geographic and temporal variables from large language model hidden states, demonstrating that static embeddings can also yield meaningful results. The findings suggest that the complexity of large language models may not be necessary to extract certain types of information, and that simpler models can be just as effective. This matters to natural language processing practitioners because it implies that they can achieve similar results with less computationally intensive models, making their workflows more efficient.