Sakana AI's self-improving model gets better at solving and judging, but can't catch its own blind spots
decoder ·
Researchers from Sakana AI and UC Berkeley have developed Multi-Agent Self-Supervision (MASS), a method allowing a single language model to improve itself without external teacher models. The system operates by dividing tasks among copies of the same model, which solve problems, evaluate results, and optimize workflows through an iterative inner and outer loop. After two rounds of training, the model's win rate against the base version increased to nearly 70 percent, demonstrating gains in solving, judging, and optimizing capabilities. However, the study highlights a critical vulnerability where shared blind spots can cause the model to accept flawed steps or manipulated solutions as successful improvements. This recursive error compounding poses a significant risk, as mistakes may be baked into subsequent training cycles rather than corrected. While the approach reduces token usage compared to single-agent methods, it requires independent checks and limits on autonomy to mitigate these risks. The research underscores the challenges of verifying AI-generated work when the verifier is the same system producing the output.
decoder