80 post-training datasets scanned. 2 flagged contaminated.

The headline finding

Of the 80 popular HuggingFace post-training datasets we scanned, 2 are measurably contaminated with public eval benchmark content (Jaccard similarity above the LQS v3.1 contamination threshold). Another 1 sit at moderate and 12 at minor — measurable n-gram overlap that doesn't yet trip the threshold but is worth disclosing to downstream model trainers.

The two flagged datasets are revealing.

• TIGER-Lab/MMLU-Pro — worst benchmark match: MMLU at similarity 0.0517. The dataset is explicitly built as a harder MMLU successor and shares stems by construction; this is expected. Worth flagging anyway because downstream model evaluators using MMLU-Pro as a "clean" MMLU alternative inherit residual MMLU n-gram overlap.
• garage-bAInd/Open-Platypus — worst benchmark match: MATH at similarity 0.1167. A widely-cited LLM fine-tuning dataset assembled from STEM benchmarks. Models fine-tuned on Open-Platypus and then evaluated on MATH are double-counting their training and eval signal.

Benchmark appearance — how often does each public eval show up in scan results?

For each of the 40+ public evaluation benchmark fingerprints indexed by the LQS contamination scanner, we counted how many of the 80 scanned datasets had any measurable n-gram overlap (similarity > 0). Top 12 below.

Interpretation: "Any overlap" means the dataset had non-zero similarity with the benchmark, not that it's contaminated. Most overlaps are below the threshold. The signal worth tracking is the worst-benchmark column in the full table below.

Full results — all 80 scanned datasets

Sorted: contaminated → moderate → minor → unknown → clean, then by max similarity descending within each tier.

Methodology

Each dataset was sampled at the listed sample size (default 600 records per split). Sampled records were tokenized into n-grams (n=8 by default for text, n=4 for code). N-gram sets were compared against the LQS benchmark fingerprint registry — a deduplicated index of n-grams drawn from the train + validation + test splits of 40+ widely-cited public evaluation suites. Maximum Jaccard similarity across all (split × benchmark) pairs is reported as max_similarity; the worst-matching benchmark is recorded.

Tier thresholds:

• clean — max similarity < 0.005, no contaminated phrase hits
• minor — max similarity 0.005–0.015 OR phrase hits below threshold
• moderate — max similarity 0.015–0.03 OR more than one minor flag
• contaminated — max similarity ≥ 0.03 OR direct phrase-match contamination on a high-stakes benchmark

The thresholds were tuned against an empirical labelled set of known-contaminated and known-clean datasets — see the empirical-threshold-validation report at validation/tools/benchmark-audit/REPORT.md.

Sample size caveat: 600 records is a representative sample, not a full enumeration. A dataset scored "clean" here may have hidden contamination in unsampled records. Re-scanning at a higher sample size or against a private fingerprint set is available via the Enterprise tier.

What's next

Contamination Report 002 will expand scope from post-training to pretraining corpora — Common Crawl, The Pile, RedPajama-V2, OSCAR, Dolma, C4, FineWeb. That's the layer where benchmark contamination most affects the score every model card reports, and where the upstream surface determines what every downstream fine-tune inherits.

If you're a maintainer interested in being scanned before the next report lands, reach out via /contact.

Back to audit series Read the methodology (DOI)