On October 27, 2026, Ai2 released a study of how reliably AI can judge whether a research idea is novel, together with a benchmark, data, code, and an automated pipeline for building new evaluation sets. The conclusion is blunt: large language models are not reliable novelty judges—change a few words in the prompt, and the verdicts on the same ideas can flip.
How the benchmark was built is what makes the study credible. The team did not treat "a paper was accepted" as a synonym for novelty. Instead, it mined ICLR 2026 peer reviews for passages where reviewers explicitly discussed originality. The high-novelty group contains 154 ideas from papers in the top 10% of their area that were accepted, where most reviewers praised originality and none disputed it; the low group contains 145 ideas from papers in the bottom 10% that were rejected, where reviewers questioned originality and none praised it. In a spot check, annotators agreed with the extracted novelty signal in 47 of 50 cases. A second comparison paired the 154 human ideas with ideas from a simple generator that did no literature search and no iteration; the team is candid that the latter carry weak labels that were not verified one by one.
The striking finding is how much the prompt moves the verdict. The study tested six judges: GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6. On the same pairs, changing only a few words—one version told the model that "reviewers found one idea novel, say which one," the other simply asked "which idea is novel"—dropped GPT-5.4's accuracy from 93.5% to 40.9%, below chance, flipping the verdict on more than half the pairs. Human-only comparisons were not immune either: the same change moved Claude Opus 4.6's accuracy by 21 percentage points. And no prompt was best for every model; helping one model could hurt another.
Adding more compute did not rescue the results. Giving GPT-5.6 Sol an extra-high reasoning budget and live retrieval produced a macro-F1 of 0.78, 11 points behind Claude Opus 4.6, at roughly 10 times the cost per decision. Two purpose-built systems did worse: Idea Novelty Checker scored 0.642 at $30.66, and Sakana AI's AI Scientist evaluator scored 0.772 at $82.05, while a simple Claude Opus 4.6 prompt baseline with three-vote majority reached 0.854 for just $2.59. The most expensive approach was neither more accurate nor more stable.
What this means for people who read papers
The study's target is not "whether AI can do science" but a sentence common in automated-ideation papers: "our system is more novel, because the LLM judge says so." If the judge itself flips with the prompt, such novelty scores are hard to treat as evidence. Ai2's advice is specific: evaluate novelty on what models will actually be asked to judge—AI-generated ideas, not only human papers; report sensitivity across prompts and models; and let benchmarks grow with the literature. The paper's second arXiv version appeared on October 14; what is new about the October 27 release is that the benchmark and pipeline are fully public. For teams building automated research workflows, the reminder is timely: first make "how to prove it is new" solid, then talk about how well the system thinks.
