MMLU — A procurement-grade audit of the most-cited LLM benchmark.

Clean parquet via the HuggingFace mirror, drop-in compatible with the datasets library. Original CSV release still available. Schema is dense: question, four options, single-letter answer key, subject tag. Multiple-choice evaluation harnesses (lm-evaluation-harness, EleutherAI's harness, OpenAI's evals) all consume MMLU without custom code. Deduction is purely for the original CSV's lack of a published schema file at release time.

MIT license. Permissive, commercial-friendly, unambiguous. No attribution chain to negotiate, no inherited license from upstream sources — questions are paraphrased or sourced from public-domain or freely-redistributable exam material. This is the model other benchmark releases should follow. The 5-point deduction is for the absence of an explicit Terms-of-Use addendum addressing the contamination question — i.e., whether maintainers expect downstream users to disclose if their training corpus may have included MMLU questions.

The Hendrycks-led group at UC Berkeley has an exceptional publication record: MMLU itself, MATH, HumanEval contributions, HARM, Hendrycks-Test, the Pile contributions. Sustained engagement with the safety / evaluation community. Responsive to errata. Multiple follow-up papers (MMLU-Pro, MMLU-Redux) acknowledge limitations openly rather than defending the original. Deduction is small and reflects the absence of an active maintenance cadence on the original MMLU repo — it's stable, not maintained.

Original paper documents the construction methodology, subject distribution, and intended use. HF dataset card lists splits, subjects, and example questions. Subject-level results from the paper are reproducible. Missing: a published label-construction protocol (how were correct answers verified?), and a published QC log. These absences are why later contamination + label-noise audits had to be done externally.

The benchmark itself is fully reproducible: the questions and answers are public, the evaluation methodology is described in the paper, multiple open-source harnesses produce consistent scores. The deduction is for the upstream sources. MMLU was assembled from "freely-available online sources" — but the exact source URLs are not catalogued in the release. Anyone wanting to verify question provenance or check whether a specific question is paraphrased from a copyrighted exam has to reconstruct that mapping themselves.

57 subjects, organized into four meta-categories (STEM, humanities, social sciences, other). Per-subject question counts vary from ~100 to ~600. Skews modestly toward STEM (formal logic, college math, abstract algebra, etc.). Some subjects are conspicuously thin: machine learning has 112 questions; college medicine has 173. For per-subject capability comparison, the smaller subjects produce ±5% confidence intervals at typical model accuracy. Acceptable for a breadth benchmark; not adequate as a per-subject diagnostic.

No published deduplication pass. Independent re-scrapes of online practice-exam sites against the MMLU corpus find a non-trivial fraction of near-duplicates — paraphrased or alternate-numbered versions of the same underlying question. Within MMLU itself, internal duplication is rare. The concern is between MMLU and the rest of the open web: the same question often exists in multiple variants, each scrapable, which feeds the contamination story below.

The paper describes a manual cleaning pass but does not publish the QC protocol, inter-annotator agreement, or rejection rate. There is no published double-annotation pass, no adjudication record, no version history of correction edits. Compare: ImageNet has a multi-rater consensus protocol with published κ; SuperGLUE publishes a multi-pass adjudication record. MMLU's labels are best-effort but not procurement-grade auditable. This is the second-largest concern after contamination.

15,908 questions total is small for a benchmark spanning 57 subjects. The aggregate composite score has tight confidence intervals (±0.4% at typical accuracies), but per-subject scores have noticeably wider ones — 95% intervals of ±4–6% on smaller subjects. For the comparison "Model A beat Model B by 0.3 points on MMLU", the sample size is adequate. For "Model A has stronger college medicine capability than Model B", the sample size is borderline. Procurement teams reading per-subject MMLU breakdowns should treat them as directional, not definitive.

Two independent re-annotation studies in 2024 found 25–30%+ of MMLU questions have at least one of: (a) no correct answer among the four options, (b) multiple defensible correct answers, (c) factually incorrect "correct" answers, (d) ambiguous wording that defeats single-answer scoring. The Hendrycks lab itself acknowledged the issue with MMLU-Redux. Importantly, this means a model scoring 87% on MMLU is, in expectation, scoring 87% on a question pool where ~28% of items have label issues. The ceiling on noise-free accuracy is somewhere around 70–75%, not 100%. Score gaps below ~3–5 percentage points between models are smaller than the label-noise floor.

One label per question (the letter A/B/C/D). No accompanying rationale, no chain-of-thought trace, no per-option explanation. This is the standard for multiple-choice benchmarks but it under-supports a procurement audit: when a model gets a question wrong, there is no labeled material to localize the failure mode. For comparison, MATH includes step-by-step solutions; HumanEval includes test-driven correctness; ARC includes rationales for the AI2 Reasoning Challenge subset. MMLU's density is procurement-relevant because it limits the kind of error analysis a buyer's evaluation team can perform.

This is the single dimension that makes the difference between Platinum and Gold. MMLU questions are scraped repeatedly across the open web: study-aid sites, AP/GRE practice forums, Quizlet decks, Anki shared collections. Common Crawl indexes most of these. Any model trained on a substantial fraction of post-2020 web text has plausibly encountered MMLU questions during pretraining.

What we can measure directly: in the LabelSets Contamination Report 001 (80 popular post-training datasets scanned against 40+ public eval benchmark fingerprints), MMLU appeared in the top-matches of 23 of 80 datasets, and TIGER-Lab/MMLU-Pro was flagged as the worst MMLU contamination case at Jaccard similarity 0.0517 — the dataset built to replace contaminated MMLU still inherits residual MMLU n-gram overlap by construction. Report 002 (in progress) will scan the upstream pretraining corpora directly.

The community fix — MMLU-Pro and MMLU-Redux — exists, but the original MMLU score is still the headline on most model cards. For procurement: an MMLU score on a model whose training corpus is undisclosed is not a contamination-clean capability signal. Substitute MMLU-Pro or a held-out subset if a clean signal is required.