Next Reply tests

A Next Reply test evaluates one Agent response at a specific conversation point. Use it for wording, policy compliance, refusal behavior, escalation, or any decision that can be judged from the next completed response.

Build valid history#

Conversation history can contain user and Agent messages plus structured Tool call and Tool result evidence. The last history entry must be a user message because the run asks the Agent to produce the next reply. History is limited to 200 entries and is frozen in the test revision used by the invocation.

You can write history manually or use Create a test from a conversation. Remove irrelevant customer data before saving a reusable test. Dynamic variables accept scalar values and should use synthetic examples for production regression suites.

Define the evaluation#

Write one concrete success condition. State what the reply must achieve, not the exact sentence it must copy. Add up to five success examples and five failure examples when they clarify tone or a subtle boundary. Examples guide evaluation; they do not replace the success condition.

The runtime combines all model output from that one Agent turn before evaluation. Streaming chunks are not scored separately. The evaluation model records status and rationale, and its token usage contributes to the run.

Interpret a result#

A failed result can mean the response missed the condition, the model request failed, quota stopped execution, or the evaluator could not return valid structured evidence. Open the individual run to read the response, evaluation, failure code, model usage, and latency before changing the Agent.

Next Reply does not execute a whole simulated dialogue and is not an interactive Test Chat. Use Simulation for multi-turn outcomes and Past runs and results for result interpretation.