Simulation tests

A Simulation evaluates an outcome across a generated multi-turn text conversation. It combines a scenario, explicit success conditions, a simulated-user model, an evaluation model, a turn limit, conversation history, and Tool-mocking policy.

Write the scenario and conditions#

Describe who the simulated user is, what they need, and the constraints they should reveal naturally. Keep facts deterministic; do not ask the simulator to invent an order state that the Agent cannot know. Add one to 30 success conditions, each describing one observable outcome. Separate policy accuracy, required disclosure, and completion behavior when they need independent evidence.

The simulator can continue for up to 50 turns; the default is five. A lower limit makes regression suites faster and more predictable. A higher limit is appropriate only when the expected path genuinely needs it.

Control the conversation#

Optional history establishes the state before simulation begins. Dynamic variables accept scalar JSON values and are frozen with the run snapshot. Procedure mentions are checked when saving and resolved again against the selected Agent configuration before a run starts, so a missing reference fails before model execution.

When no conversation history is supplied, the Agent's configured first message starts the simulation if one is present. Supplied history already establishes the opening state, so Operade does not add the configured first message again. Agent responses use the frozen production runtime, including Knowledge retrieval, Procedure and Workflow execution, and configured input or output Guardrails. The simulator decides whether to send another message or finish. The evaluator then records each success condition as structured evidence rather than changing the test definition.

Choose models and Tool behavior#

The simulated-user model and evaluation model default to gpt-5-mini. They create additional model requests beyond the Agent replies, so include them in usage planning. Evaluator failure produces a failed run; it is not silently treated as success.

Simulation uses the shared Tool-mocking policy. The safe default mocks all Tools and raises an error on a missing mock. Allowing fallback to real Tools can cause external side effects and must be an explicit review decision. Read Tool Invocation and mocks, then Attach and run tests.