NeoMME: an efficient Multimodal-native and Multilingual Encoder
huggingface ·
NeoMME is a new family of multilingual multimodal encoders (260M and 800M parameters) that processes both text and raw image patches in a single bidirectional Transformer, without a separate vision tower or causal language model. The models were pretrained from scratch using a masked discrete-diffusion objective over text, code, math, and image data. Fine-tuned as NeoMME-Retriever, it produces both dense and late-interaction embeddings in one forward pass for visual document retrieval. On the ViDoRe v3 benchmark, the 260M model achieved the highest nDCG@10 among models under 800M parameters, nearly matching ColQwen2.5 with far fewer parameters. Compression methods like hierarchical token pooling and asymmetric quantization reduced the late-interaction index size from roughly 1.5 MB to 6 kB per page while preserving over 95% of retrieval quality. The 260M model encoded about 51 pages per second on an NVIDIA L40S at 2048×2048 resolution, roughly twice the throughput of ColModernVBERT. All checkpoints are released under the Apache 2.0 license and are available in Hugging Face Transformers.
huggingface