Online Evals Track AI Agent Performance On Live Traffic Over Time

First, how are the two structurally different?

Offline, we compare an agent's behavior against a ground truth defined while building it. The eval is a dataset of inputs and expected outputs, curated to capture the behavior we want, which lets us compare performance across versions.

Online, the focus shifts from comparison to monitoring. We swap that curated dataset for live production data and score the outputs as they're generated. Two things change: we don't control the inputs (they come from real users), and we have no ground truth to measure the outputs against.

Online and offline evals aren't competitors. Rather, they tend to feed into each other. Online monitoring surfaces problematic traces, which undergo annotation and error analysis, and are ultimately converted into offline evaluations to cement behavior and capture regressions as the agent evolves. Together, they close the full evaluation loop and provide a holistic view of your agent's performance.

These constraints shape how online evals are designed. They tend to fall into two categories:

1. Heuristic evals: written as code that runs directly on the trace, measuring deterministic signals like step count, response length, or content matches.

2. LLM-as-a-judge evals: for subjective, probabilistic metrics like quality, helpfulness, hallucination, or other application-specific judgements, we score the output against a natural language rubric, i.e. a prompt.

people often only think about offline evals, but online evals are also very important!

Importantly, this lets us monitor the agent's live performance (akin to observability), see trends over time, and get alerted when a metric drops below a threshold. Combining heuristic functions and LLM-as-a-judge implementations lets us capture both hard and fuzzy metrics from live usage that may not surface in a controlled, offline experiment.

@BraceSproul The unsung hero of evals and agent observability 🦸‍♂️

@AdamRLucek Good metric split. One trap: if judge scores aren’t tied back to the original trace, you can’t debug regressions. I’d log input contract, tool outputs, heuristic failures, judge rationale, and recovery path per run so drops become replayable cases.