Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM
aws ·
AWS researchers have introduced a push-based Key-Value (KV) transfer mechanism for vLLM to enhance disaggregated LLM inference performance. This new approach improves Time-to-First-Token (TTFT) and Time-per-Output-Token (TPOT) by overlapping prefill computation with decode-side block allocation, effectively removing setup steps from the critical path. Unlike the traditional pull-based method, which requires sequential coordination after prefilling, the push mode allows the decode instance to register target blocks in advance while the producer is still computing. Benchmarks on Amazon EC2 P5en instances using the Qwen3-32B model demonstrated up to a threefold reduction in TTFT and a 30 percent improvement in TPOT under high load. The implementation utilizes a dedicated background thread to handle NIXL descriptor preparation and transfer submission, preventing latency-critical main loops from stalling. While push mode offers superior responsiveness at high request rates, it selects the decode instance earlier than pull mode, which may limit some routing flexibility. Users can enable this feature by configuring both prefill and decode servers to use the NixlPushConnector instead of the standard connector.
aws