An Anthropic researcher just gave us a peek at self-improving AI
techcrunch ·
Anthropic published a paper titled 'Automated Researchers Can Reliably Mitigate Alignment Failures,' showing AI systems that improve a model's performance on alignment benchmarks. Led by fellow Chen Yueh-Han, the system searches literature, proposes methods, and trains the model for 30 minutes per iteration, keeping effective methods. It improved performance on all 10 misaligned behavior benchmarks without degrading overall performance. The paper suggests automated alignment post-training could become practical soon, a step toward recursive self-improvement. The best automated method beats human-proposed methods on average within six hours, and costs about $4 per hour in API inference versus $150 per hour for human researchers. Limitations include dependence on benchmarks reflecting actual alignment goals and the need to maintain literature and benchmarks.
techcrunch