Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers
huggingface ·
The latest update to the Sentence Transformers library introduces native support for multi-vector, or late-interaction, embedding models through a new MultiVectorEncoder class. Unlike conventional dense embeddings that compress entire texts into a single vector, this architecture preserves individual token-level representations to maintain finer-grained semantic details. Retrieval scoring relies on the MaxSim operator, which aligns each query token with its most similar document token before aggregating the results into a final relevance score. While this method significantly boosts accuracy for complex or multi-constraint queries, it inherently demands larger storage indexes due to the increased number of vectors. The updated framework seamlessly integrates previously separate ecosystems by directly loading PyLate, Stanford-NLP ColBERT, and select visual document retrieval checkpoints via a consistent interface. Users can also leverage built-in utilities for semantic search, multimodal retrieval across images and video, token pooling for index compression, and various performance optimizations. Setting up the system requires upgrading to version 6.0 alongside compatible dependencies for transformers, PyTorch, and the Hugging Face Hub.
huggingface