It replaces unstructured and hallucination-prone secondary LLM analyses.
Why'd my agent fail? Was it reward hacking?
These days, you'd just ask another AI to vibe-analyze the agent logs
But how do you know the claims aren't hallucinated, cherrypicked, or plain wrong?
That's why we've been building Analysis Plans: a framework for trustable analysis
Users appreciate TransluceAI's analysis plans for trustable AI agent evaluations because they find the approach useful and worth thanking the team for sharing.
No Digg Deeper questions have been answered for this story yet.
the team has been cooking on a new feature for finding sus things in your agent evals!
Why'd my agent fail? Was it reward hacking?
These days, you'd just ask another AI to vibe-analyze the agent logs
But how do you know the claims aren't hallucinated, cherrypicked, or plain wrong?
That's why we've been building Analysis Plans: a framework for trustable analysis
Analysis Plans are very flexible, you can also do things like:
Understand why one agent/scaffold does better than another:
https://docent.transluce.org/dashboard/172e4804-7b43-4701-bfbf-03b974dfc4f4/analysis-plan/18fc242e-7794-495f-91f1-506f60c0e1ad
Find the most common failure modes on a benchmark run:
https://docent.transluce.org/dashboard/8ff546a5-781e-4c69-b93f-925860efb0fb/analysis-plan/8a865d08-1fbd-4578-a9f2-773d60e7d7be
Read the full blog post about Analysis Plans here: http://transluce.org/docent/blog/analysis-plans
And try Analysis Plans out for yourself! http://docs.transluce.org/analysis/quickstart
Analysis Plans is a Python framework for defining and executing analysis that’s
* Flexible: freely compose SQL steps (DQL) and LLM steps (Readings) * Verifiable: trace every result back to the computation behind it * Controllable: fine-tune details easily; it’s just code!
Without Analysis Plans, the coding agent may struggle to orient itself, cherry pick examples, overfit by reading a few non-representative transcripts, or confidently make wrong claims.
Data analysis is nuanced. You need to look at the methodology to ascertain conclusions
Each step is easy to inspect. Docent’s UI shows you the prompt template for the judge and the DQL for filtering and aggregation.
You can read the analysis plan yourself at https://docent.transluce.org/dashboard/166b1360-956d-4837-8d64-937a07518c5b/analysis-plan/86d518ee-b529-4ea8-ae32-c17e4095ccce
With Docent, the coding agent writes an Analysis Plan that’s easy to verify. In this case, the agent generated three steps:
1. Filter for successful runs 2. Score each run for suspiciousness from 0-10 with an LLM judge 3. Show the distribution of scores
Here’s an example of Analysis Plans in action.
In 15 minutes, we discovered multiple examples of a model cheating on a software engineering eval. The agent accesses nonlocal git history and copies the solution.
Are you excited about building verifiable agent analysis? We're growing our team: https://jobs.gem.com/transluce/am9icG9zdDqnxHh0KGF4UP1n5BMwIsso
@TransluceAI super useful stuff, thanks for sharing!
