the benchmark is like a tycoon game/simulation, pretty fun and probably not benchmaxxed

  • humanspiral@lemmy.ca
    link
    fedilink
    English
    arrow-up
    1
    ·
    9 days ago

    It’s a good benchmark. 3.1 is worse than 2.5 on my language test. 3 got it right, but it might have been lucky. web agent is cool. It downloaded programming language and tried running test, but bad initial knowledge doomed it to failure as it blew through all token allowance on a 5 word prompt.