AutoSynthData: Generating Training Data for Enterprise Agents
huggingface ·
ServiceNow CoreAI has introduced AutoSynthData, a system designed to generate high-quality synthetic training data for enterprise AI agents. The framework addresses the challenge of creating tasks that are feasible, realistic, and sufficiently difficult by leveraging model failures and stronger teacher models to identify capability gaps. It constructs tasks using system specifications, user prompts, and verifiers, ensuring they align with specific enterprise environments like ITSM. The process involves two phases: generating core samples and multiplying them into novel variants while maintaining quality through rigorous verification and repair loops. Experiments on the EnterpriseOps Gym benchmark demonstrate significant improvements, with the Gemma-4-26B-A4B-it model achieving a 35% relative increase in Pass@1 scores in the Hybrid domain. Similar gains were observed in the ITSM domain, validating the approach's effectiveness across different operational contexts. By continuously shifting the training frontier based on remaining weaknesses, AutoSynthData enables more targeted and effective post-training for agentic systems.
huggingface