Sentence embeddings power semantic search, clustering, retrieval-augmented generation, and many other language systems. Yet their internal structure is usually treated as opaque: we compare sentences with a single similarity score, without knowing which semantic features drive that score.
This paper develops an information-theoretic view of sentence embedding spaces. It argues that embeddings can be understood as compressed PMI-like representations, decomposable into latent semantic features and regions. The method identifies feature directions, studies their composition, and validates the resulting geometry on synthetic WordNet data and real-world TREC question categories.
The goal is to make sentence embeddings more controllable and interpretable: not just asking whether two sentences are similar, but understanding what kind of meaning they share.