Great job, this mirrors my experience running private evals nearly 1:1, down to even GPT models post Spud pre-train (so anything starting with GPT-5.5 to now) underperforming [0] [1] what one would expect based on public evals, the second one runs them privately. Also, Anthropic models have rarely under and often over-performed in my evals focused on task adherence, task evaluation, task inference and tool calling.

Love having a clear, concise and actually based in tangible information post to point to next time someone tells me that me models like Deepseek, Qwen and Grok massively underperforming what the public numbers show means I must simply not be capable enough to use them properly, even when it comes to tasks that Opus 4.5 managed but deviate from what most public evals cover.

Maybe it's not me obviously being biased against Chinese entities (Deepseeks papers are glorious and I am grateful for them being public, I also very highly rated Kimi K2 quite some time before that became more mainstream). Maybe I don't suck at prompting/context engineering/software factories/whatever this weeks hype packaging of basic dev practices is. Maybe LLMs aren't "all basically the same, only getting slightly better but never worse" even though I can get bloody GPT-6.1 Sol using the same prompt to make destructive, irrecoverable mistakes that GPT-5.0 didn't. Maybe that 300+tps inference provider actually does cut corners to get to that number and the tool calling errors and inflated tokens per task to rectify those don't actually safe money over a more reliable model, hosted by a serious, on paper more expensive, inference provider (use Deepseek if your evals show that it suites, but also eval the same model between providers, faster and/or cheaper can and often is a lot slower, less accurate and even more expensive if you take a step back).

Hope that this will finally get people to accept that there is massive training to the test by certain labs and that labs like Anthropic have shown for years that, no, this isn't inevitable and one can build models that are not just able to match but even outperform private evals [2]. Whether this is intentional or unintentional "reward hacking" (like "hallucinations", a wildly unfitting term considering is actually happening), Anthropic has shown repeatedly and over many generations, that putting in the effort, one can prevent "reward hacking"/"accidental poisoning of training data with evals"/whatever the labs excuse is.

Same thing with the Pelican SVG, shared examples often enough of that being "benchmaxxed" by Qwen and Grok [3], which perform poor on similar requests without a bike or pelican. Not sure whether it is more troubling that some labs are so insecure as to do this or that we, despite the evidence, still act like this isn't the case, but that's life.

[0] https://news.ycombinator.com/item?id=49739490

[1] https://news.ycombinator.com/item?id=49577678

[2] https://news.ycombinator.com/item?id=45597901

[3] https://news.ycombinator.com/item?id=48951229

MaxxBench – "Nice DeepSWE score. But is it true?"