How to use decision models for AI evals

In their guide, Hamel Husain and Antaripa Saha test six decision models on 100 apartment-leasing agent traces, looking for errors such as failed bookings and missed human handoffs. They warn that the default 0.50 cutoff may not suit a task and recommend tuning it to precision and recall needs before comparing performance and cost. On this dataset, OpenAI Decisions achieved the highest F1 score at the highest cost. They also recommend checking whether model scores work as confidence estimates.