How Deepgram optimized voice AI at scale with AWS Trainium
aws ·
Deepgram has successfully optimized its Nova-3 speech-to-text model for AWS Trainium2 hardware to improve inference efficiency at scale. By collaborating with AWS Forward Deployed Engineering, the company ported the PyTorch-based model to the Neuron software stack and utilized TorchNeuron for compilation. The optimization process involved replacing standard layers with fused kernels from the NKI Library and performing manual kernel tuning using Neuron Explorer and NAD tools. To maximize throughput, Deepgram implemented a data-parallel Rust runtime that eliminates Python interpreter overhead and GIL serialization issues. This configuration allows eight model replicas to run on a single trn2.3xlarge instance, significantly boosting performance for offline batch workloads. As a result, Deepgram processes 11–70 percent more audio per dollar compared to previous compute options. These improvements enable the company to serve quadrillions of audio tokens annually while reducing costs for customers self-hosting on Amazon SageMaker AI.
aws