Content Quality: 1,471 words (Analysis range 800-2000). Follows the Analysis structure: Background, Current situation (crowded field, vendor-run numbers, public indexes and a game, security test), Implications (introduced as 'not predictions', each point tied to cited reporting), and a closing 'What the sources do not establish'. Reported facts and analysis are separated; the Herald voice states no prediction or opinion. One Implications sentence ('benchmark result measured on one base model may not carry over') is a mild inference clearly framed as following from the reports. Internal links resolve: /article/2026-10/09-openai-puts-decisions-api-in-beta-... (OpenAI Decisions API beta) and /article/2026-10/12-openai-simplifies-api-usage-tiers-... (usage tiers, described accurately as a separate prior Herald article).
Source Verification: Snapshots read (gunzip + text extraction) from sources/2026-10/how-decision-models-are-being-evaluated-rival-indexes-a-pac-man-benchmark-and-a-prompt-injection-test/. source-0 (TechCrunch 2026-09-18, Tim Fernholz): confirmed 'calibrated decisions', 'cannot hallucinate', output tokens free / input metered by the billion, Vercel (Pranit Sharma) 5 to 18 times faster with greater accuracy replacing Luna, Bryo AI CTO Gemini slightly more accurate but 10-20x more expensive, Ronacher quote verbatim, Almeida 'tight-lipped' and outside observers suspect open-weight LLM. source-1 (TechCrunch 2026-10-22): confirmed Decisions API available to all customers Thursday, Luna reconfigured, Handa quote '100, 120, 180 milliseconds', 'not clear exactly what makes a good decision model, other than "it works for you"', Almeida distrusts benchmarks yet TypeSafe released its own rankings last week, AIM System One Index and AA Decision Index 'tend to favor D1 Ultra ... and Base Lab's Elephant-1', latest Jev 'in seventh and third', OpenAI 'eighth and fourteenth, respectively', OpenRouter most widely queried, JevMan PacMan benchmark built by Opper AI (Felix Wunderlich, chief growth officer), latest Jev tops it. TechCrunch gives no PacMan numbers and the article gives none; the rankings are correctly attributed to TechCrunch, not asserted as fact. source-2 (The Register 2026-10-10, Thomas Claburn): confirmed 'more than 100 such models', the entrant list, Microsoft-Decision-1 on Qwen3.5-9B, H2O-Lightning-4B on Qwen3.5-4B, 'for reasons not disclosed' rebase, 2.5x/2.8x latency, 83.5 percent on 36 benchmarks, >20x cheaper than GPT-6 Sol, $0.042 per million input tokens with free output (article dates the price to the Oct 10 report). All are Microsoft's claims relayed from its blog post and the article labels them so. source-3 (The Register 2026-10-15): confirmed HiddenLayer test (jev-ultrafast browser agent, 'Confirm Price' sponsored ad, injection worked on Qwen3.8-27B too, Jev 'succumbed less often'), quotes verbatim ('...under the hood, LLMs', 'A model that returns a decision is still a language model...'), TypeSafe did not respond; HiddenLayer is a security vendor and the article labels it as such and says the study itself was not reviewed. Scanner hit on this source reviewed in context: false positive (quotes the test's system prompt; see findings). source-4 (Artificial Analysis Decision Index page): confirmed weighted average of AA-Omniscience-Decision, HLE-MultiChoice, When2Call, WildGuardTest (25% each), 'All evaluations are conducted independently by Artificial Analysis', cost/median time per task, 'Decision models are billed for input tokens only...', 'Calibration for language models coming soon.' The snapshot lists model names (incl. Jev 1.14, GPT-6 Luna (Decisions), Microsoft-Decision-1) but no readable numeric scores or ranks, so TechCrunch's placements cannot be confirmed from it, exactly as the article discloses; the page shows no date. source-5 (OpenAI Decisions guide): confirmed 'returns typed answers about 10x faster than the Responses API' and 'gpt-6-luna, the only currently supported model'. Live re-fetch of TechCrunch Oct 22, the AA page and the OpenAI guide on Oct 25 matched the snapshots (WebFetch of the AA page reported a visible leaderboard; the raw snapshot lacks the values, so nothing was added from it). INDEPENDENT-SOURCE COUNT (Analysis requirement): 2 firmly independent outlets (TechCrunch, The Register; 4 articles), 4+ sources total. TechCrunch did original reporting (interviews with Almeida, Handa, Ronacher, Vercel, Bryo, Speak, Opper). The Register largely relays vendor material (Microsoft blog, HiddenLayer study) but labels it and adds context (the 'more than 100' count, the Qwen bases, the TypeSafe non-response). Artificial Analysis is counted only as a conditional third: it is a third-party benchmark publisher with no stake in the vendors, but it is also one of the rival indexes the article analyses, so it is treated as an object of analysis and not relied on for the count. OpenAI's guide is vendor documentation and is not counted. Requirement (at least 2 independent) is met without Artificial Analysis. All 6 URLs are in article.sources with no orphan body URLs; allowlist hosts techcrunch.com, theregister.com, artificialanalysis.ai and developers.openai.com are present by exact hostname (no allowlist edits).
Factual Accuracy: No fabricated or misattributed claim found. Rankings, Microsoft numbers and HiddenLayer findings are consistently attributed to TechCrunch, Microsoft and HiddenLayer. 'Decision model' usage (typed or probabilistic outputs for classification and routing) matches the Herald's earlier Decisions API article. One omitted tension: The Register describes Jev as 'a large language model (LLM) tuned to respond...' while TechCrunch says Jev 'is not a large language model'; the article attributes the 'not an LLM' wording to TechCrunch and quotes HiddenLayer's 'under the hood, LLMs' but does not state that the outlets disagree. Recorded as a recommendation, not a correction, since no statement is wrong.
Overall Assessment: Meets the Analysis requirements (1,471 words; 6 sources; 2 independent outlets without relying on Artificial Analysis; examines a topic; facts and analysis separated; limits section present). Approved as an Analysis with no corrections needed. Featured candidacy: 1 - accurate cross-source look at how a new model category is benchmarked, of interest mainly to developers evaluating decision models