Benchmarking small LLM inference on SageMaker AI: G7 vs G5 and G6
aws ·
This article benchmarks small Large Language Model inference on Amazon SageMaker AI, comparing the new G7 instances powered by NVIDIA Blackwell GPUs against previous generations like G5 and G6. The study evaluates two representative 30B Mixture-of-Experts models to demonstrate how G7 delivers measurable gains in throughput, latency, and cost-per-token. In the first use case involving an AI coding assistant, researchers deployed Qwen3-Coder-30B across ml.g5.12xlarge, ml.g6.12xlarge, and ml.g7.12xlarge instances using the DJL Large Model Inference container. Results indicated that G7 achieved approximately 60.8 percent higher output token throughput than G6 and reduced average request latency by 37.6 percent compared to G6. Streaming performance tests on G7 showed a median time to first token of roughly 118 milliseconds and consistent inter-token latency, making it suitable for interactive developer tools. The second use case utilized Amazon SageMaker AI Generative AI Inference Recommendations with vLLM to optimize configurations for an enterprise AI assistant model. This automated workflow evaluated candidate instance families to identify the best price-performance options based on specific workload objectives. Key hardware advantages of G7 include native support for NVFP4 quantization and higher memory bandwidth, which significantly benefits memory-bound MoE architectures during token generation. Ultimately, the benchmarks illustrate that while real-world gains depend on model architecture and workload shape, G7 offers superior performance for these specific generative AI deployments.
aws