Benchmark contamination means that the questions or answers used in an evaluation were already mixed into the training data before the model was trained. The model is not solving the problems on the spot; it is answering from memory. Its score looks high, but that does not prove it has really mastered the skill.
How contamination happens
The most common route is direct: a public benchmark gets collected into training corpora. Once test questions are published, crawlers can pick them up from web pages, papers, code repositories and Q&A posts, and they end up in the model along with huge amounts of other text. A second route is indirect leakage: questions are rewritten, translated or broken up into discussion posts, with the answers hidden inside explanations, so the model remembers the same question in a different form. A third route happens during evaluation itself: the same set of questions is used again and again to tune settings and pick models, and the score slowly gets tuned upward until the benchmark stops working as a referee.
An inflated score does not mean the ability is fake
Benchmark contamination is easily misread as the model cheating. In most cases it is a data-pipeline problem, not a model deliberately memorising answers. The real problem is that the meaning of the score changes: it no longer measures generalisation, but how much of the test the model has memorised. A model that scores very highly on one set of maths questions may drop noticeably on a newly released set of similar questions it has never seen. Conversely, a modest score does not mean the model is useless; some capable models have simply never seen that benchmark's question style.
Four questions to ask before trusting a score
First, how new are the questions: was the benchmark released recently, and was it deduplicated against the training data? Second, is the evaluation transparent: does the report explain the deduplication method and publish contamination checks? Third, is there more than one line of evidence: does the same ability hold up across benchmarks from different sources and in different formats? Fourth, look at real tasks: try the model on a small piece of a real workflow and watch how it performs on data it has never seen. That tells you more than a leaderboard alone.
Two common misunderstandings
One misunderstanding is equating benchmark gaming with contamination. Optimising for a benchmark's format or choosing the best prompt is legitimate test preparation; only training data that contains the test itself counts as contamination. The second misunderstanding is thinking this only affects open models. Closed models face the same problem with benchmark questions in public corpora; the difference is only that outsiders find it harder to check their training data. When reading a model announcement, always read the score together with the evaluation method. A single number on its own says very little.
