Shreya Shankar details a custom AI eval workflow that uses Claude's Monitor tool to steer coding agents
The workflow replaces generic automated benchmarks with domain-expert failure taxonomies.
Story Brief
The workflow replaces generic automated benchmarks with domain-expert failure taxonomies.
Related Stories
2 linkedHamel's NewsletterI write about topics like evals, fine-tuning, and infrastructure for LLMs.6h agohamel.dev
How to Automate AI Evals (Correctly)Join the AI Evals September 2026 cohort: https://maven.com/parlance-labs/evals?promoCode=yt-2026 Most automated eval tools find surface level problems or struggle with taste. But the failures that actually matter live in the product team's head. Shreya Shankar is a CS professor and co-creator of the AI Evals course on Maven. In this session, she covers three mistakes teams make when automating evals and demos a live workflow that uses AI to scale human judgment. Timestamps: 00:00:00 Introduction 00:00:55 Why vendors want to automate evals for you 00:01:45 Why no tool can fully automate evals 00:02:44 The analyze, measure, improve lifecycle 00:06:00 Mistake #1: Asking AI to just "find the issues" in your data 00:08:11 What good AI-assisted error analysis actually looks like 00:11:12 Live demo: building a review interface from scratch 00:15:10 Annotating traces and building a failure mode taxonomy 00:19:31 Mistake #2: Only reviewing your data once 00:22:53 Mistake #3: Treating all apps with the same accuracy bar 00:25:10 Surprising finding: general-purpose agents vs. dedicated eval tools 00:25:48 Summary: three rules for automating evals correctly Connect with Hamel: Website ► https://hamel.dev LinkedIn ► https://www.linkedin.com/in/hamelhusain/ Twitter/X ► https://x.com/hamelhusain Instagram ► https://www.instagram.com/hamelsmu/ Tik Tok ► https://www.tiktok.com/@hamel_husain Connect with Shreya: Website ► https://www.sh-reya.com LinkedIn ► https://www.linkedin.com/in/shrshn
How to Automate AI Evals (Correctly)Join the AI Evals September 2026 cohort: https://maven.com/parlance-labs/evals?promoCode=yt-2026 Most automated eval tools find surface level problems or struggle with taste. But the failures that actually matter live in the product team's head. Shreya Shankar is a CS professor and co-creator of the AI Evals course on Maven. In this session, she covers three mistakes teams make when automating evals and demos a live workflow that uses AI to scale human judgment. Timestamps: 00:00:00 Introduction 00:00:55 Why vendors want to automate evals for you 00:01:45 Why no tool can fully automate evals 00:02:44 The analyze, measure, improve lifecycle 00:06:00 Mistake #1: Asking AI to just "find the issues" in your data 00:08:11 What good AI-assisted error analysis actually looks like 00:11:12 Live demo: building a review interface from scratch 00:15:10 Annotating traces and building a failure mode taxonomy 00:19:31 Mistake #2: Only reviewing your data once 00:22:53 Mistake #3: Treating all apps with the same accuracy bar 00:25:10 Surprising finding: general-purpose agents vs. dedicated eval tools 00:25:48 Summary: three rules for automating evals correctly Connect with Hamel: Website ► https://hamel.dev LinkedIn ► https://www.linkedin.com/in/hamelhusain/ Twitter/X ► https://x.com/hamelhusain Instagram ► https://www.instagram.com/hamelsmu/ Tik Tok ► https://www.tiktok.com/@hamel_husain Connect with Shreya: Website ► https://www.sh-reya.com LinkedIn ► https://www.linkedin.com/in/shrshnCommentary on X
Highest ranked
Tech News@tech_summariesTECH
the "unknown unknowns" part is the real unlock here. most teams waste time on generic metrics that don't capture product taste. the shift from "automated evaluation" to "human-in-the-loop error analysis" is the only way to actually ship reliable agents. that live demo in the session is gold for anyone still stuck on vibe checks.
