Fancy harnesses make coding agents more expensive but not better, study finds
decoder ·
A recent study by researchers at Shandong University challenges the effectiveness of complex coding agent harnesses, finding that they increase costs without improving performance. The paper, titled "What does a harness buy? Mostly tokens," reveals that swapping between major frameworks like Claude Code and mini-SWE-agent yields results statistically equivalent to simply rerunning the same setup. While some vendors claim significant score gains, these differences often fall within the natural noise range of benchmark tests. The primary impact of fancy harnesses is a substantial increase in token consumption, driven largely by lengthy system prompts and tool descriptions rather than better task resolution. Only one harness, OpenCode, showed a measurable performance deficit, which was partly attributed to output limits and configuration issues. The study highlights that model choice has a far greater impact on outcomes than the specific harness used. Consequently, the authors argue that current leaderboards may be misleading due to uncontrolled variables and lack of repeated trials. They recommend publishing reruns alongside comparisons to distinguish genuine improvements from random variance.
decoder