An advanced guide to AI evaluations and what to automate
Lenny’s Newsletter says its new guide draws on the authors’ work with 50-plus AI companies, covering what to automate, what not to automate and a free plugin that lets coding agents help with the work.
TLDR
Lenny’s Newsletter shares an advanced guide to AI evaluations, covering a key step it says most teams skip and a free plugin it says lets a coding agent do most of the heavy lifting. In its September 22 announcement, the newsletter said nearly half of the 25 product-manager job openings it shared the week before asked for eval-writing experience. It also cites companies’ reported gains from investing in evals: Ramp increased automatic receipt collection accuracy from 35% to 83%, while Shopify shipped an AI workflow builder described as 2.2 times faster and 68% cheaper than the frontier-model setup it replaced.
