Impactful scheduling for GPU clusters
huggingface ·
The Allen Institute for AI (Ai2) replaced its priority-based GPU scheduler with a budget-driven system to address resource scarcity and inefficiencies like 'squatting' and priority inflation. By allocating GPU time through hierarchical fair-share budgets, the institute shifted decision-making from operational negotiations to transparent administrative planning. The new scheduler incorporates a 'scheduling contract' that requires workloads to declare minimum runtimes, allowing for automatic preemption and requeuing once progress is secured. Simulations predicted significant improvements in queue latency, which were validated during production rollout where debug workload wait times dropped from hours to seconds. Operational metrics showed that teams received 98% of their allocated GPU hours while maintaining 98% cluster occupancy. Additionally, automated handling of preempted jobs reduced human-in-the-loop maintenance tasks by 74%. Despite these successes, the team identified challenges regarding interactive session state preservation and potential capacity fragmentation for large workloads. Future efforts will focus on optimizing utilization and supporting dev sessions via dedicated CPU clusters.
huggingface