Spreading the load: How Salesforce met Multi-AZ HA with SageMaker Inference Components
aws ·
Salesforce used the new IC Placement capability in Amazon SageMaker AI Inference Components to meet multi-AZ high availability requirements for Agentforce. Default placement optimized costs but could unevenly distribute model copies, creating single points of failure at the instance or AZ level. The SchedulingConfig parameter, with AvailabilityZoneBalance and PlacementStrategy (SPREAD or BINPACK), provides fine-grained control over copy placement across AZs and instances. By setting MaxImbalance and ensuring CopyCount of at least two, Salesforce spread copies across AZs to eliminate single points of failure. They also used On-Demand Capacity Reservations and SageMaker AI Insights for capacity planning and monitoring AZ balance. As a result, every model deployment satisfied their 2-AZ compliance bar while preserving the 8x cost reduction from co-hosting models on shared GPUs. The article recommends designing HA at the inference component level and maintaining balance during scaling and model updates.
aws