Simplify and support your TorchServe workloads using Ray Serve Deep Learning Containers
aws ·
TorchServe is no longer actively maintained, so teams using it must handle security patches, compatibility updates, and dependency management themselves. AWS now offers Ray Serve Deep Learning Containers (DLCs), pre-built and tested images that bundle PyTorch, Ray Serve, the GPU stack, and utilities for multimodal workloads. The blog post demonstrates deploying the Qwen3-VL-2B vision-language model on Amazon EKS using a single g5.xlarge GPU node and the Ray Serve DLC. The serving application is written as a Python class with Ray Serve decorators, replacing TorchServe's handler, model archiver, and configuration file. The deployment uses a ConfigMap for the serving code and includes scripts to provision the cluster, node group, and Ray Serve deployment. Users can test the endpoint by port-forwarding and sending a curl request with an image URL and prompt. The post recommends this as a migration path from TorchServe, noting that KubeRay can extend the setup to multi-node distributed serving.
aws