Can we trust LLMs to judge whether a research idea is novel?
huggingface ·
Allen Institute for AI has released a new study and benchmark titled 'Old Ideas, Novel Problems' to evaluate the reliability of large language models acting as novelty judges for scientific research. The research reveals that LLMs struggle to consistently distinguish novel ideas from existing work, with small changes in prompts causing dramatic shifts in verdicts. In experiments involving six major models, altering prompt wording changed accuracy by over 50 percentage points when evaluating AI-generated ideas. While judges appeared more stable on human-authored papers, they exhibited significant brittleness when assessing outputs from automated ideation systems. Furthermore, adding complex retrieval mechanisms or higher reasoning budgets did not improve performance and often increased costs without enhancing accuracy. Purpose-built evaluation pipelines performed worse than simple prompted baselines, highlighting inefficiencies in current approaches. These findings suggest that automating the validation of scientific novelty remains an unresolved challenge requiring more robust and standardized evaluation methods.
huggingface