Reduce inference cold starts on Amazon SageMaker HyperPod with model caching
aws ·
AWS announced model caching for Amazon SageMaker Inference on HyperPod, a feature designed to reduce the cold-start delay that occurs between requesting an inference pod and the pod being ready to serve traffic. The delay is caused by two sequential downloads: the inference server container image from Amazon ECR and the model weights from sources such as Amazon S3, Amazon FSx for Lustre, or HuggingFace Hub, which can take 30 minutes or more for large models like DeepSeek-R1 at over 600 GB. Model caching addresses this with two independent capabilities: a weights cache that pre-loads model weights onto local NVMe storage on cluster nodes, and an image cache that pre-pulls the container image so pods skip the ECR pull entirely. Weights cache labels nodes as cache-ready and blocks deployment creation until all target nodes are ready, while image cache does not block deployment creation. Both use preferred rather than required scheduling, so pods are never blocked and fall back to normal network downloads if a node lacks a warm cache. Benchmarks show roughly 60 percent faster scale-out with weights caching and up to a 97 percent reduction in cold image-pull time compared with pulling fresh from ECR. Enabling the feature requires only adding a modelCacheConfig section to an existing InferenceEndpointConfig or JumpStartModel resource, and the operator handles cache lifecycle and cleanup. The feature is generally available in all regions where SageMaker HyperPod is available, though it requires sufficient NVMe capacity and source updates are not auto-detected.
aws