Chapter 08 of 12 · Aishwarya Ashok
Peanut checks the work and reads the receipts
Did the job succeed, was the route acceptable, and what did it consume? Evaluate the outcome alongside the steps that produced it. A successful example is a starting point for measurement.
Systematic tests that assess an agent's behavior and results against defined criteria.
An evaluation suite pairs representative tasks with criteria for judging behavior and results. Graders may be code, people, or models. Include failures that matter in the intended use, and repeat trials where run-to-run variation is significant. The suite becomes useful engineering feedback when its results point to something the team can investigate or improve.
The proportion of evaluated tasks or runs that satisfy the defined success criteria.
For this illustrative test, seven successes out of ten evaluated deliveries gives a 70% success rate. Be explicit about the denominator: tasks, runs, and attempts can produce different measures. Also state the criteria and how retries count. That information lets a reader interpret the percentage and compare it fairly with another result.
Assessing the sequence of an agent's actions and observations, rather than only its final output.
The delivery succeeded, but the route violated a constraint. A final-output check would miss that.
Trajectory evaluation examines actions and observations for behavior such as missed approvals, unsuitable tools, or wasteful loops. Allow valid alternative routes unless the task specifically requires one sequence.
A measure of whether an agent's tool requests use appropriate tools and arguments for the task.
Evaluate the selected tool and its arguments against the task's requirements. This separates a mistaken request from an execution failure at the service. A valid call to the wrong record is a tool-call problem even if the service reports success. Clear scoring criteria help identify which part of selection or argument construction needs work.
Re-running established checks after a change to detect behavior that has worsened or broken.
Changes to a prompt, model, tool, or retrieval setup can disturb behavior that used to work. Re-run established checks to catch that deterioration. Keep the test conditions comparable and use repeated trials where necessary to account for variability. A stable evaluation set gives the team a shared basis for discussing whether the new version is better.
Recording connected execution events so a request or run can be inspected across its steps and services.
Tracing connects recorded operations from the same request or run. Model calls, tool results, timings, and errors become easier to investigate when their relationships are preserved. Instrumentation determines what is visible. Use a trace to reconstruct observed execution, without treating it as a complete account of the model's internal processing.
The ordered sequence of actions, observations, and relevant decisions made during a particular run.
The trajectory is the path a run took through actions, observations, and relevant decisions. A trace can supply the records needed to reconstruct it. Comparing trajectories helps explain why similar requests produced different outcomes. Sometimes the difference is a tool failure; sometimes it is a different decision made from the available context.
Recorded units of work within a trace, each with timing and contextual information.
A span records a unit of work with timing and contextual attributes. Parent-child relationships show operations performed within another operation. Concurrent spans can overlap, so adding their durations may exceed total elapsed time. This structure helps distinguish a slow tool from time spent elsewhere in the larger request.
An allowance or limit on token usage for a defined model call, task, or period.
Set the scope of the allowance first: one call, a whole task, or an accounting period. Then specify how input, output, and reasoning usage count toward it.
A task budget can cover many model calls. The context window constrains a single inference context; the token units themselves are explained in Tokens and tokenization.
The elapsed time between a defined start event and the corresponding response or completion event.
Time to the first output and time to task completion measure different experiences. Pick the interval that answers your question, then account for model calls, queues, tool waits, and retries within it. A faster initial response may coexist with a slow overall workflow. Name the metric when comparing versions or describing performance to users.
The cost attributed to completing or attempting one task under a stated accounting method.
Decide which expenses belong to a task before calculating its cost. Token charges, tools, compute, and human review may all be relevant. Cost per attempt and cost per successful task tell different stories when failures are common. Use a consistent accounting boundary so a seemingly cheaper approach is not just leaving part of the bill elsewhere.
Worked example · Order 17 · installment 3 of 4
A successful dispatch answers only part of the evaluation. The reviewer also checks whether the agent used the correct details, obtained approval at the right time, and avoided a duplicate after the timeout. The recorded trajectory supplies evidence for those checks.
This run belongs in a broader suite with missing records, ineligible requests, and denied approvals. Repeating the suite after changes helps detect regressions. Timing and cost records answer a different question: what it took to complete the work.
Concepts together: Agent evals, trajectory evaluation, tracing, regression testing, latency, cost per task.
Next in the replacement-order example →