Recent math benchmarks for large language models (LLMs) such as MathArena indicate that state-of-the-art reasoning models achieve impressive performance on mathematical competitions like AIME, with the leading model, o3-mini, achieving scores comparable to top human competitors. However, these benchmarks evaluate models solely based on final numerical answers, neglecting rigorous reasoning and proof generation which are essential for real-world mathematical tasks. To address this, we introduce the first comprehensive evaluation of full-solution reasoning for challenging mathematical problems. Using expert human annotators, we evaluated several state-of-the-art reasoning models on the six problems from the 2025 USAMO within hours of their release. Our results reveal that all tested models struggled significantly, achieving less than 5% on average. Through detailed analysis of reasoning traces, we identify the most common failure modes and find several unwanted artifacts arising from the optimization strategies employed during model training. Overall, our results suggest that current LLMs are inadequate for rigorous mathematical reasoning tasks, highlighting the need for substantial improvements in reasoning and proof generation capabilities.
“Notably, O3-MINI, despite being one of the best reasoning models, frequently
skipped essential proof steps by labeling them as “trivial”, even when their validity was crucial.”
“Notably, O3-MINI, despite being one of the best reasoning models, frequently skipped essential proof steps by labeling them as “trivial”, even when their validity was crucial.”
LLMs achieve reasoning level of average rationalist
This is actually an accurate representation of most “gifted olympiad laureate attempting to solve a freshman CS problem on the blackboard” students I’ve went to uni with.
Jumps to the front after 5 seconds from the task being assigned, bluffs that the problem is trivial, tries to salvage their reasoning for 5 minutes when questioned by the tutor, turns out the theorem they said was trivial is actually false, sits down having wasted 10 minutes of everyone’s time.
I just remember a professor saying that after he filled the board with proofs and math. ‘the rest is trivial’ not sure if it was a joke, as I found none of it trivial. (and neither did the rest of the people doing the course).
it’s a very human and annoying way of bullshitting. I took every opportunity to crush this habit out of undergrads. “If you say trivial, obvious, or clearly, that usually means you’re making a mistake and you’re avoiding thinking about it”
LLMs achieve reasoning level of average rationalist
This is actually an accurate representation of most “gifted olympiad laureate attempting to solve a freshman CS problem on the blackboard” students I’ve went to uni with.
Jumps to the front after 5 seconds from the task being assigned, bluffs that the problem is trivial, tries to salvage their reasoning for 5 minutes when questioned by the tutor, turns out the theorem they said was trivial is actually false, sits down having wasted 10 minutes of everyone’s time.
I just remember a professor saying that after he filled the board with proofs and math. ‘the rest is trivial’ not sure if it was a joke, as I found none of it trivial. (and neither did the rest of the people doing the course).
This needed a TW jfc (jk, uh, sorta)
TW: contains real chuds
“Trivially” fits nicely in a margin, too. Suck on that, Andrew and Pierre!
it’s a very human and annoying way of bullshitting. I took every opportunity to crush this habit out of undergrads. “If you say trivial, obvious, or clearly, that usually means you’re making a mistake and you’re avoiding thinking about it”
feels like the same manner as my “‘just’ is a weaselword” speech