Deploying Qwen3.8-2.4T-A95B on Amazon SageMaker HyperPod with vLLM
aws ·
The article explains how to deploy Alibaba's Qwen3.8-2.4T-A95B, the first open-weight model in the Qwen-Max class, on Amazon SageMaker HyperPod using vLLM. This model has 2.4 trillion total parameters with only 95 billion activated per token, features a hybrid linear-plus-full-attention architecture, and supports a native context of up to 262K tokens, making it suitable for demanding agentic and reasoning workloads. The deployment uses a single ml.p6-b300.48xlarge instance with eight NVIDIA B300 GPUs, relying on NVFP4 (W4A4) quantization to compress the 2.4T weights to roughly 1.2 TB and fit within the available GPU memory. SageMaker HyperPod is highlighted for its EKS-orchestrated clusters, Inference Operator, automatic model weight downloads, scaling, and node replacement, easing the operational burden of hosting trillion-parameter models. The vLLM serving configuration includes tensor parallelism across eight GPUs, prefix caching, a Qwen3 reasoning parser, auto tool-calling support, and native Multi-Token Prediction (MTP) for speculative decoding. The post also covers infrastructure sizing, memory budgeting, and key vLLM flags, offering a complete path from cluster setup to an OpenAI-compatible endpoint.
aws