Epoch AI 审计 Humanity's Last Exam：抽样 48 题中 46% 存在影响准确率的错误，判定为 Flawed

Humanity’s Last Exam is a knowledge-based benchmark compiled through submissions from the public. After auditing a random sample of 48 questions, we found that 22 (46%) questions had substantial accuracy-altering errors, leading us to designate the benchmark as Flawed.

Methodology

We reviewed the original benchmark, Humanity’s Last Exam, not Humanity’s Last Exam-Rolling or Humanity’s Last Exam-Verified. We randomly sampled 48 questions, 6 from each of 8 categories (Biology/Medicine, Chemistry, Computer Science/AI, Engineering, Humanities/Social Science, Mathematics, Physics, Other), then prompted Fable 5 to surface errors, primarily of three kinds: questions that are essentially impossible to answer correctly as written (impossible), questions that could mark an objectively correct answer incorrect (false negatives), and questions that could mark an incorrect answer correct (false positives). We audited and confirmed each error by hand, filtering out identified errors that did not meet our evidence bar.

Of those 48 questions, we found that 12 were impossible to answer correctly as written, for reasons like missing assumptions never given to the LLM, contradictory premises, multiple valid interpretations, or questions that demanded objective answers to subjective questions. An additional 10 well-posed questions could produce false negatives, of which 5 could also produce false positives. In total, 22 of 48 questions, or 46%, were defective.

Representative Errors

Computer Science/AI: Impossible. The question asks for a “4-point DFT”, a discrete Fourier transform over 4 samples, but the sequence has 8 entries.
Engineering: False negative, false positive. The author’s answer key ends with the formula printed in option E, but the answer key says D.
Humanities/Social Science: False negative. The question asks for a three-letter abbreviation for 1 Kings. Both “1kin” and “1kgs” are commonly used abbreviations, but the answer key matches only “1kin”. “1kgs” is graded as incorrect.
Mathematics: False negative, false positive. The author’s rationale calculates the normalization constant as 1/21.3535, but the answer key is 21.35.

Errors

Task	Category	Error Type	Notes
6736c2e82db9d8cc7cd8e607	Biology/Medicine	Impossible	Medical case study already states the confirmed diagnosis, but asks for what the next diagnostic step is
66ff35926045e9e64bbc5720	Chemistry	Impossible	Asks for density as a function of height, but answer key is a single number
66f275c6e2a2b5d3594eae87	Computer Science/AI	Impossible	Asks for the smallest possible memory, but answer key’s program is provably not the smallest
672f6cfd609b665328565e5c	Computer Science/AI	Impossible	The most memory-efficient program has no defined minimum, answer key’s 66 bytes can be beaten
66f1384dba7cd382e52dc0c6	Computer Science/AI	Impossible	Question asks for a 4-point DFT but the sequence has 8 entries
66f2298b868e51658baf900a	Engineering	Impossible	Data supports several defensible values, key silently ignores one listed input
67200b2f79e871b075407673	Engineering	Impossible	Problem says the circuit is matched, then deducts a quantity that doesn’t need to be deducted if the circuit is matched
67aacfd513ec9e1a16359d51	Engineering	Impossible	CPU-cycle count depends on how you measure
671fadb2272218e5d336a33f	Engineering	Impossible	No answer option is fully correct
6737224020f2fc6d8d08075c	Humanities/Social Science	Impossible	Asks about the themes of the author’s own unpublished artwork
671683479e5d26a2905120dc	Other	Impossible	Intended solution requires a mob that cannot spawn close to the player under normal conditions, and an alternate block also works under abnormal conditions of the intended solution
672333955d82e15ca8e37afb	Physics	Impossible	Answer changes depending on which uncertainty-principle convention is used, and none is specified
67371496e04511118435d5a4	Computer Science/AI	False negative, false positive	Key’s time-dilation factor has a misplaced decimal (0.9963 vs 0.963)
671fee7b3fdf9b50dcb7ca58	Engineering	False negative, false positive	Mislabeled answer: the author’s own rationale matches option E, but key is D
6711e5e05e64a53ed09449fd	Humanities/Social Science	False negative, false positive	Answer key (0.218) comes from rounding mid-calculation, exact answer is 0.220
67381ce26a5242a22fe4681f	Mathematics	False negative, false positive	Answer key gives the reciprocal of the answer found in its own rationale
67361730dba36cc0d595f422	Physics	False negative, false positive	Answer key cites some interference that doesn’t occur
6734b98afaf4e6eb454c1c9d	Biology/Medicine	False negative	Answer key misspells the organism’s genus name and LLM judge may thus mark it wrong
67253e40cff9fdccf85f3f08	Chemistry	False negative	Multiple molecules also satisfy every listed constraint, but only one is in the answer key
6730f3c006cd33fe46ca2dfe	Chemistry	False negative	Answer key’s mass value is slightly off at the demanded precision
67190f1eee94d36fc7734311	Humanities/Social Science	False negative	Key uses a nonstandard Bible-book abbreviation that could easily be graded incorrectly due to difference in abbreviation
6724f79b262cac26e3c38360	Humanities/Social Science	False negative	Only one Unicode rendering of the same word is accepted in the answer key

The answer keys of the other 26 sampled questions were found to be correct, with id values as follows: 668825f80a642802bdfeadfa, 66e4cdec11c64a7e4051b2d9, 66e883265ab37f0a7da089be, 66eaa5ddc7a3252f0f3fe53f, 66eaf9bbb082c5e6a76a49b5, 66eb3aab27b19dd593af7ebe, 66ecfd03e7478ca5fcbb7495, 66f02cc56ea864e80ba2d3d7, 66f042e57cd17eb270eb9ccb, 66f864c8f7c10f2d2ba396df, 66fea0c4cb66b0e85c55ee52, 67029d8de7ea2ce5def2f39f, 6705b23880154ff607a205b5, 670be48d7038d6936230870a, 6713a7d01e0e03ffe2253dd0, 67142e7e7da71e9cbf55a7f9, 67178b22b7f6dd9be2a4dd68, 671ad72c9fdc33c08a784b3a, 671ee4d26edc2afd69958953, 67216f0abddbccdce9955e93, 6722b1f9f3a489ee08117730, 672455ce7a14971607e3d9f0, 672588e1f71812e186947615, 672fadd93c2722c42adabef3, 67378905b73b79a4008926af and 674982572b1759933c70c4eb.

Rubric

1. Reviewability

Level	Meaning	Status
Full	All tasks and scoring logic inspectable*, and harness/API settings used for each model (reasoning effort, token/time limits, tool access, system prompts) are fully disclosed	[proceed to 2]
Partial	A representative sample of tasks and scoring logic inspectable, with full harness/API settings disclosed	[proceed to 2]
Inadequate	Limited, biased, or no inspectable tasks or scoring logic, harness/API settings undisclosed	[stop → NEI]

* Either publicly or privately to reviewers.

2. Scoring

This is the minimum standard required to be Verified; failing any of these items results in a Flawed verdict.

Scoring
Examples	Essentially impossible to correctly answer as written (e.g. underspecified task, hidden requirement, missing file/tool) False Negative (e.g. overly strict scorer, stale/incorrect ground truth, dependent on live external state that can drift, sandbox failure independent of the agent) False Positive (e.g. lax scorer, reward-hackable environment, stated skill can be bypassed via a shortcut such as exploiting an error in the scoring logic or retrieving the answer from the harness/web) Task egregiously doesn’t measure the claimed capability
Default threshold for Flawed	≥20% of inspected sample† contains errors or there is an issue that corrupts grading at scale
Status	PassFlag [stop → Flawed]Not reviewed
Notes	—

Benchmark Consistency
Examples	Scorer, instructions, or ground truth changed without a version bump
Default threshold for Flawed	Leaderboard has incomparable results from different versions
Status	PassFlag [stop → Flawed]Not reviewed
Notes	—

Elicitation
Examples	Model elicitation is extremely constraining and is not the focus of the benchmark: Under-resourced relative to task size (token/turn/time limit, sandbox resources) Poor context management Lack of agentic environment where it would be natural to provide one Excessive non-voluntary termination for agentic benchmarks
Default threshold for Flawed	Substantially reduced performance compared to reasonable alternatives for the tasks
Status	PassFlag [stop → Flawed]Not reviewed
Notes	—

Bias in Evaluation Setup
Examples	Uneven compute/token budgets Unfair scaffold choice (e.g. only a subset of models optimized)
Default threshold for Flawed	Material model-specific advantage found
Status	PassFlag [stop → Flawed]Not reviewed
Notes	—

† For benchmarks with >50 available tasks, we will sample a random set of 50 (stratified by category, when present). If the observed error rate is 15–25%, we will expand the sample to 100. For benchmarks with ≤50 available tasks, all available tasks will be assessed.

3. Evaluation Quality

This is the standard we would like all benchmarks to meet, but it is not necessarily disqualifying to omit or fail these items.

Question	Status	Notes
Elicitation and resource adequacy: Are the resources given to models (reasoning token/turn budget, tool access, etc.) sufficient for them to perform near their ceiling?	SufficientConstrainingUnreasonably constrainingUnknownNot reviewed	—
Scaffold fairness: What scaffold does the leaderboard report?	Shared common scaffoldMix of model-specific and common scaffoldsModel-specific scaffoldsNot reviewed	—
Is there evidence/risk of contamination?	Not reviewed	—
Has human completability been assessed?	All tasksRepresentative set of tasksPoor implementation (Unrepresentative set of tasks, unreasonable set of participants)Not establishedNot reviewed	—
Score range (if possible to estimate)	— floor, — ceilingNot reviewed	—
Statistical adequacy: how many runs/model (≥ 5 recommended for error bars)	— runs/modelUnknownNot reviewed	—
Construct Validity	Measures stated capabilitiesPartially measures stated capabilitiesDoes not measure stated capabilitiesNot reviewed	—