LLMs are nearing the limits of what they can do; the 2025-style innovation leap is not repeating.

Hypothesis Log

2026-05-21 v1
LLMs are nearing the limits of what they can do; the 2025-style innovation leap is not repeating.
Opening hypothesis — intentionally broad. Will be refined as evidence lands. — read post

Evidence

Same Opus 4.8 release: Anthropic itself led with 'a modest but tangible improvement' framing — endorsed by Simon Willison as 'refreshing.' Within-family benchmark deltas are incremental: SWE-Bench Verified 87.6% → 88.6% (+1.0pp); SWE-Bench Pro 64.3% → 69.2% (+4.9pp); Terminal-Bench 2.1 still goes to GPT-5.5 at 78.2% vs Opus 4.8 74.6% (no Claude-side breakaway); OSWorld-Verified near-flat against an Anthropic-restated 4.7 baseline (zoom-tool bug fix). Headline benchmark Anthropic chose to lead with was honesty (4x fewer unflagged code errors vs 4.7) — a reliability/UX metric rather than capability per se. Within-family-incremental pattern is consistent with a polish-cycle release inside an established family rather than a step-change frontier leap.
2026-05-28 · Anthropic Opus 4.8 announcement; Vellum (restated baseline analysis); Every.to (Willison endorsement quote); The Decoder · post →
Claude Opus 4.8 released May 28 2026 — 41 days after Opus 4.7 (April 16), fastest Opus cycle ever (Opus 4.6 → 4.7 was ~7 months). Three Opus releases in three calendar months — densest Anthropic frontier release schedule on record. Independent measurements: Artificial Analysis Intelligence Index 61.4 (Opus 4.7: 57.3; GPT-5.5: 60.2) — first time since OpenAI's April launch a Claude model holds #1 spot; GDPval-AA 1890 Elo (+137 vs Opus 4.7, +121 vs GPT-5.5, ~67% head-to-head win rate); Browserbase Online-Mind2Web 84% (independent eval, Miguel Gonzalez); Harvey Legal Agent Benchmark first model to break 10% at all-pass standard; Cursor: more efficient tool calling than 4.7. Anthropic raised $65B at $965B valuation alongside launch. Mythos-class GA hinted at 'in the coming weeks' (preview was 93.9% SWE-Bench Verified, 100% Cybench — held back on cybersecurity not capability). 3-6 month frontier SWE-Bench Verified rate ~+2pp/month matches or modestly exceeds the preceding 12-month rate; cadence acceleration is the clearest counter-signal to 'innovation leap is not repeating' this cycle.
2026-05-28 · Anthropic Opus 4.8 announcement; Artificial Analysis Intelligence Index; LLM-Stats blog; Vellum benchmarks summary; Browserbase Online-Mind2Web (Gonzalez); Harvey Legal Agent Benchmark; Every.to Vibe Check; SiliconAngle ($65B raise) · post →
LMArena top tier (GPT-5.4 / Claude Opus 4.6 Thinking / Gemini 3.1 Pro / Grok 4 / DeepSeek V3.2) has stayed in the 1,450-1,561 Elo band over the last 6 months; Arena Expert methodology sharpened in November 2025, which compresses top-cluster scores mechanically. Consistent with plateau but methodology change makes the signal genuinely ambiguous on Elo data alone over a 6-month window. Counter-signal on the same axis: SWE-Bench Verified frontier moved from ~65% in Q1 2025 to 88.7% vendor / 82-83% independent (vals.ai) by April 2026 with GPT-5.5, roughly 1-2 percentage points per month at the frontier; Claude Mythos preview at 93.9% on at least one tracker; SWE-Bench Pro (contamination-resistant successor) in the 55-65% range with substantial remaining headroom — these argue against plateau on the broad axis, so the verify-direction evidence is real but weak.
2026-05-20 · LMArena leaderboard arena.ai; LMArena changelog (Arena Expert Nov 2025); vals.ai SWE-Bench Verified independent leaderboard; LocalAI Master SWE-Bench history; BenchLM Elo history · post →
OpenAI internal general-purpose reasoning model autonomously disproved Erdős 1946 planar unit-distance conjecture; peer-verified May 20 by 9-author arXiv companion paper (Alon, Bloom, Gowers, Litt, Sawin, Shankar, Tsimerman, Wang, Wood); Sawin separate paper gives explicit delta=0.014 via class field towers / Golod-Shafarevich theory; Gowers (Fields medalist) characterised as Annals of Mathematics quality; Bloom (maintainer of erdosproblems.com, prior critic of OpenAI math claims) co-authors the verification. Sits inside an existing ~18-month cadence of peer-verified narrow autonomous-discovery events at frontier labs (AlphaProof IMO silver Jul 2024 + Nature methodology Nov 2025; IMO 2025 gold from DeepMind Deep Think and OpenAI; AlphaEvolve arXiv 2511.02864 Nov 2025 improving bounds across 67 problems; FunSearch cap set; PatternBoost). Cleanest credential panel of the genre to date but consistent with the existing rate, not a break in it.
2026-05-20 · arXiv 2605.20695; arXiv 2605.20579 (Sawin); OpenAI announcement; AlphaEvolve arXiv 2511.02864; AlphaProof Nature Nov 2025 · post →

All posts on this hypothesis