Graham Neubig and researchers release PACE, a proxy method predicting LLM agent performance at 100x lower cost
The framework achieves a 3.8% prediction error across 14 models.
Story Brief
The framework achieves a 3.8% prediction error across 14 models.
Source Posts on X
Cluster sources
Graham Neubig@gneubigTECH
Agentic evaluations are sloooow, and expeeeensive. Can we instead get a good idea of how well an LLM will do on agentic tasks by running it solely on single-turn tasks? Our method PACE comes up with an effective strategy for doing so! https://twitter.com/yueqi_song/status/2074180763302670648
CLS@ChengleiSiTECH