Measuring LLM judge accuracy after model changes
A post recommends checking an LLM judge’s accuracy every time its own model changes, drawing a parallel with annual reviews of trading-surveillance systems.
TLDR
An LLM judge—a model used to evaluate AI outputs—should have its accuracy measured whenever its own model changes, one post argues. It cites chapter 5 of Evals for AI Engineers on estimating true success rates with imperfect judges. The comparison comes from MiFID II RTS 6, Article 13(6): the post quotes a requirement for investment firms engaged in algorithmic trading to review automated surveillance at least annually, including its ability to minimise false positive and false negative alerts.
Combined views
976
2 Sources, first seen ago
