Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM
aws ·
AWS has introduced a bi-directional key-value (KV) cache transfer mechanism in vLLM to optimize multi-turn agentic workloads. This feature addresses inefficiencies in prefill-decode disaggregated serving by allowing decode nodes to return previously computed KV blocks to prefill nodes for subsequent conversation turns. By reusing this context, the system avoids redundant recomputation of growing conversation histories, which significantly reduces the time to first token (TTFT). Benchmarks on Amazon EC2 P5en instances using AWS Elastic Fabric Adapter demonstrated TTFT reductions of up to 1.6x for Qwen3-32B and 3.3x for zai-org/GLM-5-FP8 at long prompt lengths. The implementation relies on a KV-cache-aware router to manage metadata and configurable thresholds for block retention and recomputation decisions. Originally validated on AWS Trainium, this accelerator-agnostic capability is now supported on GPU clusters via the NIXL KV connector enhancements. Users can enable this feature in vLLM version 0.21.0 or later to sustain responsive performance as agent interactions scale.
aws