tokenizers v1: encode, decode and scaling, measured
huggingface ·
Hugging Face has released a release candidate for tokenizers v1, aiming to resolve performance bottlenecks in machine learning workflows as models scale. The new version achieves encoding speeds three to thirty times faster than v0.23 by replacing regex-based splitting with SIMD-powered bitstream operations via the Bitcannon library. Key optimizations include a thread-local word cache that memoizes repeated pre-tokens and a rewrite of the merge loop to eliminate memory allocations using scratch buffers. These changes ensure that GPUs remain active without waiting for CPU tokenization, significantly improving throughput for large datasets and concurrent requests. The update maintains full API and output compatibility with previous versions while supporting multiple tokenizer families like BPE, WordPiece, and Unigram. Benchmarks conducted on Apple M4 Max hardware demonstrate linear scaling across eight worker threads, validating the effectiveness of native parallelism. Future plans involve expanding model family support and exploring GPU-accelerated encoding in subsequent releases.
huggingface