Epoch AI 评审 SWE-Bench Pro：超 20% 任务存在评分缺陷，判定为 Flawed

SWE-Bench Pro was meant to be a more challenging and contamination-resistant successor to SWE-Bench Verified. However, audits of SWE-Bench Pro have estimated that 30% or more of tasks are broken, leading us to designate this benchmark as Flawed.

Methodology

SWE-Bench Pro consists of 1,865 tasks: 731 are public, 276 are private, and 858 are held-out. The tasks were sourced from actively maintained repos and are meant to simulate long-horizon professional software engineering workflows.

Multiple audits have been performed on the public set:

On February 24th, 2026, Jonathan Gabor audited 100 random SWE-Bench Pro problems and found 83 had issues.¹ The main error type was “requirements inflation” - implementation details specified in the requirements section were not tested.
On May 26, 2026, Datacurve performed an LLM-assisted audit and found 24% false negatives and 8.5% false positives.²
On July 8th, 2026, OpenAI published an audit estimating 30% of tasks were broken.³ After an initial datapoint analysis pipeline, each flagged task was independently reviewed by five experienced software engineers. Errors mainly fell into the following four categories: misleading prompts, overly strict tests, underspecified prompts, and low-coverage tests.

Given these results, it seems very likely that more than 20% of questions contain scoring defects, thus we designate this benchmark as Flawed.

In addition to these scoring defects, Poolside documented numerous reward hacking behaviors in the task environment.⁴

Acknowledgements

This review relies on published audits by OpenAI, Jonathan Gabor, Datacurve, and Poolside.

Rubric

1. Reviewability

Level	Meaning	Status
Full	All tasks and scoring logic inspectable*, and harness/API settings used for each model (reasoning effort, token/time limits, tool access, system prompts) are fully disclosed	[proceed to 2]
Partial	A representative sample of tasks and scoring logic inspectable, with full harness/API settings disclosed	[proceed to 2]
Inadequate	Limited, biased, or no inspectable tasks or scoring logic, harness/API settings undisclosed	[stop → NEI]

* Either publicly or privately to reviewers.

2. Scoring

This is the minimum standard required to be Verified; failing any of these items results in a Flawed verdict.

Scoring
Examples	Essentially impossible to correctly answer as written (e.g. underspecified task, hidden requirement, missing file/tool) False Negative (e.g. overly strict scorer, stale/incorrect ground truth, dependent on live external state that can drift, sandbox failure independent of the agent) False Positive (e.g. lax scorer, reward-hackable environment, stated skill can be bypassed via a shortcut such as exploiting an error in the scoring logic or retrieving the answer from the harness/web) Task egregiously doesn’t measure the claimed capability
Default threshold for Flawed	≥20% of inspected sample† contains errors or there is an issue that corrupts grading at scale
Status	PassFlag [stop → Flawed]Not reviewed
Notes	—

Benchmark Consistency
Examples	Scorer, instructions, or ground truth changed without a version bump
Default threshold for Flawed	Leaderboard has incomparable results from different versions
Status	PassFlag [stop → Flawed]Not reviewed
Notes	—

Elicitation
Examples	Model elicitation is extremely constraining and is not the focus of the benchmark: Under-resourced relative to task size (token/turn/time limit, sandbox resources) Poor context management Lack of agentic environment where it would be natural to provide one Excessive non-voluntary termination for agentic benchmarks
Default threshold for Flawed	Substantially reduced performance compared to reasonable alternatives for the tasks
Status	PassFlag [stop → Flawed]Not reviewed
Notes	—

Bias in Evaluation Setup
Examples	Uneven compute/token budgets Unfair scaffold choice (e.g. only a subset of models optimized)
Default threshold for Flawed	Material model-specific advantage found
Status	PassFlag [stop → Flawed]Not reviewed
Notes	The leaderboard combines runs with a capped cost limit and turn limit of 50 (greyed out) and runs with an uncapped cost and with a turn limit of 250.

† For benchmarks with >50 available tasks, we will sample a random set of 50 (stratified by category, when present). If the observed error rate is 15–25%, we will expand the sample to 100. For benchmarks with ≤50 available tasks, all available tasks will be assessed.

3. Evaluation Quality

This is the standard we would like all benchmarks to meet, but it is not necessarily disqualifying to omit or fail these items.

Question	Status	Notes
Elicitation and resource adequacy: Are the resources given to models (reasoning token/turn budget, tool access, etc.) sufficient for them to perform near their ceiling?	SufficientConstrainingUnreasonably constrainingUnknownNot reviewed	—
Scaffold fairness: What scaffold does the leaderboard report?	Shared common scaffoldMix of model-specific and common scaffoldsModel-specific scaffoldsNot reviewed	—
Is there evidence/risk of contamination?	Not reviewed	—
Has human completability been assessed?	All tasksRepresentative set of tasksPoor implementation (Unrepresentative set of tasks, unreasonable set of participants)Not establishedNot reviewed	—
Score range (if possible to estimate)	— floor, — ceilingNot reviewed	—
Statistical adequacy: how many runs/model (≥ 5 recommended for error bars)	— runs/modelUnknownNot reviewed	—
Construct Validity	Measures stated capabilitiesPartially measures stated capabilitiesDoes not measure stated capabilitiesNot reviewed	—

Disclaimer

This is a partial review. We stopped inspecting once we discovered errors that exceeded our published threshold for a flawed benchmark.

We attempted to reach out to the benchmark developer prior to launch, and if they write a response to this review we will link it here. If you’re the benchmark developer, please email us at reviews@epoch.ai with a link to your response if you would like us to include it here.

About Benchmark Reviews

Epoch AI’s Benchmark Reviews are independent reviews of external AI benchmarks. Our documentation describes the rubric behind this verdict, how we choose which benchmarks to review, and answers frequently asked questions.

Go to source Read the documentation and FAQs