LangSmith Engine Automates AI Agent Failure Analysis and Fixes

VIEWS222RETWEETS1

Fun chat with @hwchase17 talking about our work on Engine!

Special episode with @bentannyhill on the Max Agency podcast. A month ago, his team shipped LangSmith Engine, our agent that hunts through your agent's failures, prioritizes issues, and drafts the fix.

We dive into the architecture decisions -- from how we used sandboxes to how we created subagents. And we also discuss an interesting challenge -- how we were able to build evals for an agent that never stops running.

Check out the full conversation ⤵️

⏯️ YouTube: https://youtu.be/YqjR4vQwbTc?si=wp2AKnhbVnFSCzb8

🎧 Apple: https://podcasts.apple.com/nz/podcast/the-best-ai-agents-are-secretly-teams-ben-tannyhill/id1891551672?i=1000775182821

🟢 Spotify: https://open.spotify.com/episode/5lT8hva20sVmJ0WiVCLAaO?si=81119f1d193e4f7d

5h1.6K71

LangChain@LangChain

🎧A look at how we built LangSmith Engine with @hwchase17 + @bentannyhill

Harrison Chase@hwchase17

Special episode with @bentannyhill on the Max Agency podcast. A month ago, his team shipped LangSmith Engine, our agent that hunts through your agent's failures, prioritizes issues, and drafts the fix.

We dive into the architecture decisions -- from how we used sandboxes to how we created subagents. And we also discuss an interesting challenge -- how we were able to build evals for an agent that never stops running.

Check out the full conversation ⤵️

⏯️ YouTube: https://youtu.be/YqjR4vQwbTc?si=wp2AKnhbVnFSCzb8

🎧 Apple: https://podcasts.apple.com/nz/podcast/the-best-ai-agents-are-secretly-teams-ben-tannyhill/id1891551672?i=1000775182821

🟢 Spotify: https://open.spotify.com/episode/5lT8hva20sVmJ0WiVCLAaO?si=81119f1d193e4f7d

6h5.8K283

milly.hl@milly_rock7

@LangChain @hwchase17 @bentannyhill agent-agnostic or LangSmith-loyal sounds like the real debate here

feels like they learned all the lessons so we dont have to

5h3

Eclipse 🌖@ECLresearch

@hwchase17 @bentannyhill Sandbox-driven debugging shifts the bottleneck from issue discovery to triage — curious how the prioritization engine weighs failure severity vs. frequency in practice.

6h3

Jett Monroe 🐧✳️@JettMonroe0x

@LangChain @hwchase17 @bentannyhill honestly the idea of an agent that hunts agent failures is lowkey terrifying and brilliant at the same time

your agents own quality assurance agent

5h2

katamannas@WaelAlsaeed9

@LangChain @hwchase17 @bentannyhill Most teams adapt existing frameworks by adding time-windowed metrics. Instead of pass/fail per run, they measure drift, latency trends, and recovery rates over rolling periods.

5h

marygrace_1713@nana_tourSVT

@LangChain @hwchase17 @bentannyhill The real challenge isn't the framework—it's defining what "failure" means in a system that never stops. Traditional eval assumes clear start/stop points. Continuous agents need a different mental model.

5h