Stop guessing auto scaling targets for TTS on Amazon SageMaker AI
aws ·
This article presents a method for eliminating guesswork in auto-scaling targets for Text-to-Speech (TTS) endpoints on Amazon SageMaker AI. It highlights that default metrics often fail for TTS because requests hold connections open without token-generation loops, making concurrency the critical factor. The author deploys the Kokoro-82M model and measures two key values: Q_max, the maximum concurrency before Service Level Objective breach, and T_total, the end-to-end scale-out time. Using these measurements, the article derives precise scale-out and scale-in thresholds along with asymmetric cooldowns to prevent flapping. The solution uses AWS Cloud Development Kit stacks and a custom CLI tool chain to ensure no unmeasured configuration reaches production. This approach replaces assumptions with data-driven arithmetic, ensuring cost efficiency and latency compliance. The methodology is designed to be generalizable to other latency-sensitive, real-time endpoints beyond just TTS.
aws