Accelerate agentic AI with NVIDIA Nemotron models on Amazon EKS and SageMaker HyperPod EKS
aws ·
AWS and NVIDIA have announced the availability of the Nemotron 3.5 Super VL model on Amazon EKS and SageMaker HyperPod EKS, expanding the existing lineup of Nemotron models on AWS. This new 120B-parameter mixture-of-experts model activates only 12B parameters per token, combining reasoning with text, image, document, and video understanding in a single multimodal architecture. The model utilizes a hybrid Mamba-Transformer design that supports context lengths up to one million tokens while maintaining efficient memory usage through constant-size state layers. It is designed for agentic AI workflows, including coding assistance, web research, document intelligence, cybersecurity threat triage, and video analysis. Deployment options include standard Amazon EKS, EKS Auto Mode, and SageMaker HyperPod EKS, utilizing NVIDIA Dynamo and vLLM for inference serving. Performance validation has been conducted on B200 and H200 GPU instances, demonstrating capabilities for both aggregated and disaggregated deployment topologies. The article provides detailed guidance on capacity planning using latency metrics like time to first token and inter-token latency to optimize throughput within service level objectives.
aws