ProgramBench has since patched the network bypass loophole.
Agents on ProgramBench reimplement software, with no internet access. Sonnet 4.6 realized it's in a benchmark, then found a clever way to bypass our internet restriction. This and more fixed in the latest release 🧵
No Digg Deeper questions have been answered for this story yet.
must read blog, some of the most creative cheats i have seen
modern benching is increasingly becoming whack-a-mole against agents exploiting the shit out of everything
Agents on ProgramBench reimplement software, with no internet access. Sonnet 4.6 realized it's in a benchmark, then found a clever way to bypass our internet restriction. This and more fixed in the latest release 🧵
Agents on ProgramBench reimplement software, with no internet access. Sonnet 4.6 realized it's in a benchmark, then found a clever way to bypass our internet restriction. This and more fixed in the latest release 🧵
The blocking of internet access while running the agent worked fine, however, we did run our evaluation scripts _with_ internet. Sonnet 4.6 was just guessing this might happen and put download commands into its submission to download during evaluation (rather than inference)
This and more explained in the blog: https://programbench.com/blog/release-1-1-0/
ProgramBench is open source at https://github.com/facebookresearch/ProgramBench/
ProgramBench is a joint effort across Meta FAIR, Meta TBD, Stanford, Harvard
@jyangballin (co-first author) @18jeffreyma @parth007_96 @dpedch @sten_sootla @micmylin @pengchengyin @magpie_rayhou @syhw @Diyi_Yang @OfirPress
