Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference
aws ·
Amazon SageMaker Inference has introduced prefix-aware routing, a new routing strategy that inspects the beginning of each request and consistently directs requests with the same prefix to the same instance. This matters because LLM prompts typically contain a large fixed portion, such as instructions or retrieved documents, followed by a small variable user input, and serving frameworks like vLLM and TensorRT-LLM can cache the key-value pairs for those prefixes. Without prefix-aware routing, requests are spread across a fleet too thinly for any instance to build a useful cache. In benchmarks on Llama 3.1 70B Instruct across seven ml.p5.48xlarge instances with vLLM, the feature reduced P50 time-to-first-token by up to 77 percent, increased throughput by up to 16 percent, and raised KV cache hit rates from roughly 25 percent to over 80 percent. The endpoint includes safeguards for overload protection, routing to a less busy instance when a target is at its configured concurrency limit, and stable behavior during scaling so caches are not invalidated. SageMaker now offers three routing strategies—RANDOM, LEAST_OUTSTANDING_REQUESTS, and the new PREFIX_AWARE—configurable per production variant and switchable without redeploying the model. It is most useful for RAG applications, multi-turn conversations, templated bots, and code completion, and two parameters, PrefixLength and ConcurrencyThreshold, control its behavior. No changes to the model container or serving framework are required, and multi-tenant isolation can be achieved with an optional prefix-aware ID. The article recommends enabling prefix caching in the serving framework, keeping request serialization consistent, sizing PrefixLength carefully, using at least two instances, and monitoring cache hit rates.
aws