Right-size generative AI endpoints with concurrency sweeps on Amazon SageMaker AI
aws ·
This article introduces concurrency sweeps as a systematic method for right-sizing generative AI endpoints on Amazon SageMaker AI to maximize price-performance while maintaining acceptable latency. By sending controlled, increasing levels of concurrent traffic, users can identify an endpoint's saturation point where throughput plateaus and latency spikes. The workflow involves deploying models using the native vLLM container, configuring workload profiles, and running automated benchmarks via the CreateAIBenchmarkJob API. Key metrics such as throughput, p99 latency, and time to first token are analyzed to determine optimal capacity and prevent over-provisioning or service degradation. For more advanced planning, the article describes an automatic SLA-based search recipe that efficiently finds the maximum concurrency satisfying specific performance thresholds. This approach replaces manual guesswork with data-driven decisions, allowing teams to confidently size their fleets before production traffic arrives. The process is illustrated through a walkthrough of the NVIDIA Nemotron-3 Nano 30B model, demonstrating how to interpret results and clean up resources afterward.
aws