Reactions from ranked influencers
3 posts
IssueBench is a detailed evaluation suite we built for Engine (continual learning agent in LangSmith, runs on your traces, automatically improves your agent) Evaluating Engine was a tricky problem due to the type of data it needs to run on, so we wrote this blog outlining what the benchmark is, and how we made it:
IssueBench is our internal benchmark for evaluating Engine (a continual learning agent in LangSmith) This blog by @nick_bray dives into why we needed it, what it tests and how we built it
LangSmith Engine is our in product agent that runs over traces, clusters them into issues, and proposes fixes It's a long running, complex, ambiguous process We wrote about how we evaluate it!