Run multi-node inference workloads using Ray LLM Deep Learning Containers
aws ·
This article demonstrates how to deploy large language models across multiple GPU nodes on Amazon EKS using the new Ray LLM Deep Learning Containers. The solution implements prefill and decode disaggregation, where one node handles prompt processing while another generates tokens, thereby preventing resource contention between compute-heavy and memory-bound phases. AWS provides a pre-built container image that bundles vLLM, Ray Serve, NIXL for cache transfer, and the Elastic Fabric Adapter stack, eliminating the need for custom boilerplate code or manual dependency management. The deployment architecture utilizes a CPU head node for orchestration and two GPU worker nodes pinned to a single Availability Zone to ensure low-latency communication via EFA. Configuration is simplified through a YAML-based application builder that automatically wires the KV-cache transfer behind an OpenAI-compatible endpoint. The guide outlines the step-by-step process for provisioning the EKS cluster, installing the KubeRay operator, and deploying the RayService with specific model parameters. Finally, it provides instructions for invoking the endpoint, verifying EFA usage, and cleaning up resources to avoid ongoing charges.
aws