Introducing Amazon SageMaker HyperPod Inference Gateway
aws ·
AWS announces Amazon SageMaker HyperPod Inference Gateway, a Kubernetes-native, GPU-aware routing addon for existing HyperPod and EKS clusters that requires no application or model-server changes. Its two-tier design includes an Envoy Gateway, a Body-Based Router for model selection, and an Endpoint Picker that uses real-time metrics such as KV-cache utilization, queue depth, LoRA adapter residency, prefix-cache hits, and active requests. The gateway can be installed as a single EKS addon and configured declaratively, while exposing a standard OpenAI-compatible endpoint for existing clients. It supports multi-model routing and directs LoRA requests to pods that already have the required adapter loaded. AWS benchmarks against Kubernetes round-robin routing reported substantial tail first-token latency reductions and throughput gains in mixed-GPU, bursty-traffic, and shared-prefix scenarios, while results were comparable on uniform fleets under steady traffic. Tier 1 is available in regions that support the inference addon, while the cross-cluster and cross-region Global Inference Router is planned for a future release.
aws