Reduce ASR inference costs by 75% with NVIDIA MPS on Amazon EC2
aws ·
AWS, NVIDIA, and Heidi Health collaborated to reduce automatic speech recognition (ASR) inference costs on Amazon EC2. ASR requests typically use only 15–20% of a GPU's compute capacity, and CUDA's default time-slicing leaves much of the hardware idle, forcing Heidi to run 16 GPU instances. The solution combines NVIDIA CUDA Multi-Process Service (MPS) with NVIDIA Triton Inference Server on EC2 GPU instances, cutting GPU infrastructure needs by 75% from 16 instances to 4. This setup maintains sub-second latency while handling 92.1 requests per second per GPU. The pipeline uses a FastAPI gateway, Triton for dynamic and sequence batching, and MPS to partition the GPU into concurrent execution contexts. Model-level optimizations include running the Conformer encoder through ONNX Runtime with TensorRT, while the RNN-T decoder runs in PyTorch CUDA. An accompanying open-source repository provides Dockerfiles and configuration for deploying the solution on g6e.4xlarge or g7e.4xlarge instances with NVIDIA L40S GPUs.
aws