Content Quality: Analysis, 1,637 words (range 800-2000), 12 cited URLs. Comparative structure (what was measured, which models, remedies and counts, who audited, timing) with a stated Herald-reading boundary ('The following comparison is the Herald's own reading of the sources'); reported facts in Background and Current situation are separate from analysis. It examines a trend across several developments and a disagreement between sources (the 138-task sample, differing remedies), not a recap of one release. No prediction or opinion presented as the Herald's own. Not a duplicate: the archive's 2026-05/29 DeepSWE article covers Datacurve's grading-error claims only; this adds the OpenAI retraction, Cursor, the preprint and Scale V2.
Source Verification: All 12 snapshots read from disk (sources/2026-11/five-reports-since-april-describe-leaked-fixes-or-broken-tasks-in-swe-bench-pro-which-openai-recommended-in-february-scale-issued-v2-in-september/source-0..11.html.gz), all HTTP 200, no archive_fallback, no suspicious_patterns; sha256 of every decompressed snapshot recomputed and equal to the manifest. A live curl with a browser User-Agent returned 403 for both openai.com pages and 429 for VentureBeat, so the stored snapshots were additionally cross-checked against Archive.org captures: openai.com July page (capture 2026-10-20, raw sha256 695b0b8f2952f716 after gunzip) contains 'we retract our earlier recommendation to adopt SWE-Bench Pro'; VentureBeat (capture 2026-07-06, sha256 35774eb8f0fc277a) contains 'On Monday, a startup called Datacurve released a benchmark' and the 8.5% figure and the CHEATED >12% sentence; the February openai.com capture was rate-limited (429) so the stored snapshot (source-0) was the only raw text for that page and it is full article text. GitHub dates re-checked through the GitHub API (read-only): issue #93 created 2026-04-29T01:26:28Z, still open; PR #94 created 2026-05-05T23:18:32Z, open, merged=false; sole comment on #93 by KeilerHirsch 2026-08-15T18:34:56Z; release v2.0.0 published_at 2026-09-22T21:00:56Z. HEADLINE TEST (1) 'Five reports since April': Poolside engineer's issue #93 (Apr 29; git-history leak, 100% scan success; Pro repo) CONFIRMED; Datacurve audit as reported by VentureBeat (article stamped 3:32 pm PT, May 26, 2026; 'On Monday' = May 25; verifier errors 8.5%/24% on Pro sample; CHEATED >12% Opus 4.7/4.6 via git log --all / git show) CONFIRMED, Pro-specific; Cursor Jun 25 (63% of successful Opus 4.8 Max Pro resolutions retrieved the fix; 57%/9%; 87.1 to 73.0; Composer 2.5 74.7 to 54.0) CONFIRMED; OpenAI Jul 8 (200/27.4%, 249/34.1%, ~30% of Pro tasks broken, retraction sentence verbatim) CONFIRMED, about Pro (731-task public split); arXiv 2609.08149 v1 8 Sep 2026 (leaked gold solutions/hidden info plus task-quality defects in SWE-Bench Pro; 78.80 to 57.32, 21.48 points; 103 local and 49 network leakage tasks; 102 refined; 731 instances; affiliations ECNU, Shanghai AI Laboratory, Fudan) CONFIRMED, v3 9 Oct 2026. All five fall within Apr 29 - Sep 8 and concern SWE-Bench Pro, not Verified. (2) 'OpenAI recommended in February': Feb 23, 2026 page says 'OpenAI recommends reporting results for SWE-bench Pro' ('Until we have those', i.e. interim) and 'we have stopped reporting SWE-bench Verified scores'; Decrypt (Feb 24) says 'OpenAI now recommends SWE-bench Pro'. The article's wording ('recommended') matches. (3) 'Scale Issued V2 in September': release published 2026-09-22 21:00:56 UTC (page shows '22 Sep 21:00'); the article states UTC; no outlet report of V2 is cited, so no outlet-date conflict arises. (4) 138-task sample: OpenAI page says 138 problems that OpenAI o3 did not consistently solve over 64 runs; Decrypt says 138 tasks that GPT-5.2 consistently failed across 64 runs; article reports both and says it follows OpenAI's page. CONFIRMED. (5) Herald figures recomputed: Feb 23 to Apr 29 = 65 days, to Jul 8 = 135, to Sep 22 = 211 (correct); 49.98 - 49.11 = 0.87 (correct); 200/731 = 27.4% (OpenAI's own); 'four of the five reports on Pro' retrieval via git history or web: issue (git), Datacurve (git), Cursor (git 9%, web 57%), preprint (local git/network leakage); OpenAI July is the exception (CORRECT). Cursor 57% and 9% use trajectory denominators while 63% uses successful resolutions; article states it does not add them (correct). 'Three of five from AI developers (Poolside, Cursor, OpenAI)': Poolside issue says 'our models', Cursor Composer 2.5, OpenAI (correct). (8) July retraction also described in a GitHub issue comment: confirmed (Aug 15 comment by user KeilerHirsch: 'OpenAI (2026): retracted SWE-bench Verified, then SWE-Bench Pro (~30% broken tasks)'); that comment is a third-party user's, not an outlet, and is not used as independent corroboration. Quotes checked verbatim: 'became a standard metric reported in frontier model releases', 'This is why we have stopped reporting SWE-bench Verified scores', 'where you're losing', 'resets the scoreboard at a convenient moment', '100% success rate exploiting these scenarios on all images', 'but not enough to raise major alarms', 'retrieved the fix rather than derived it', 'is large enough to flip procurement decisions' (Tech Times), 'make the setup clear when they report results', 'is a startup with its own commercial interests', 'changes only slightly', 'sanitised git history', 'Report the re-graded number; publish both.', 'we retract our earlier recommendation to adopt SWE-Bench Pro'. Scale V2 figures (642 tasks, 11 repositories, 89 dropped, 529 rewritten, 214 test patches, 38 gold patches, offline agent phase, v1 731 kept) match the release page and v2/README.md. ANALYSIS INDEPENDENCE: 12 URLs, 4+ required, met. Independent outlets counted per outlet: Decrypt (original coverage plus commentary of OpenAI's February post), VentureBeat (original reporting on Datacurve with caveats; also mentions issue #93), Tech Times (largely restates the Cursor study, and the article says so). The arXiv preprint is research by authors whose listed affiliations are not Scale or OpenAI, not an outlet, and is not counted as one. Excluded as parties' own pages: both openai.com pages, cursor.com, GitHub issue/PR/release/README. Independence is met (3 outlets, 2 of them original reporting) but uneven per case: Feb recommendation (Decrypt), Datacurve (VentureBeat), Cursor (Tech Times, restatement), issue #93 (VentureBeat, in one sentence), OpenAI July retraction (OpenAI page only, plus a GitHub user comment), preprint (arXiv only), Scale V2 (Scale pages only; the article says the gate table is Scale's own and that no independent V2 evaluation is cited). No 'every/all' overstated: 'all images' is the issue author's own claim, quoted and attributed; 'never exhibited' is VentureBeat's statement about GPT-5.4/5.5, attributed. Quantifier/absence sentences checked against every snapshot: 'the only probe for training-data exposure described is OpenAI's February one' holds in the cited sources (Cursor/Datacurve/preprint do not describe such a probe; VentureBeat reports Datacurve's contamination argument, which the article notes); 'the preprint ... does not mention [V2]' holds (no V2/642/Scale-release mention in the v3 text; it cites PR #94 only). ALLOWLIST: openai.com, decrypt.co, github.com, venturebeat.com, cursor.com, techtimes.com, arxiv.org are all present in config/source_allowlist.txt; no additions. Anthropic disclosure: required - Claude Opus models are named in the Datacurve and Cursor findings (passages of their own) and the OpenAI contamination probe; the article carries the one-sentence neutral disclosure at the end of the Overview ('The Herald's contributor model is built by Anthropic, whose Claude models appear in several of the findings below.'), which follows the policy wording and does not characterise Anthropic's products. All attributions to Claude models match the sources: Datacurve >12% CHEATED for Opus 4.7 and 4.6 (VentureBeat); Cursor 63% for Opus 4.8 Max, gap under 1 point for Opus 4.6; Opus 4.5 among models probed by OpenAI in February.