Usage, safety, and limits

Agent Testing can multiply model requests quickly. Treat suite size, repeats, Simulator and evaluator calls, and real Tool policy as release controls rather than convenience settings.

Hard limits#

• A saved definition is at most 256 KiB after canonical JSON encoding.
• Test and folder names are 1 to 200 characters; whitespace-only test names are rejected.
• Conversation history is at most 200 entries.
• conversation_initiation_source is either unknown or telegram_integration.
• A Simulation has 1 to 30 success conditions and 1 to 50 turns.
• Next Reply accepts up to five success examples and five failure examples.
• A parameter path and regex pattern are each at most 500 characters. Regex evaluation has a 100 ms timeout and a timeout is a non-match.
• One library bulk move accepts 1 to 100 entities.
• Test-library and invocation-summary pages accept 1 to 100 rows.
• Invocation run pages accept 1 to 1,000 rows.
• One Agent configuration can contain at most 5,000 attached_tests references.
• One invocation expands to at most 5,000 test entries and accepts 1 to 50 repeats.
• The maximum expansion is therefore 250,000 runs.
• One resubmit accepts 1 to 5,000 unique run IDs.

The UI offers smaller repeat presets, but the API limit remains 50. Start with one repeat while authoring, then increase only when variance is part of the release question.

Model usage and quota#

Agent replies consume model usage. Simulation also uses a simulated-user model and evaluator; Next Reply uses an evaluator; repeated-run failure bucketing can use another evaluator. A request rejected by activation or quota during preflight returns HTTP 403 or 429 and does not create the invocation. Quota reached later in a worker is stored on the affected run as quota_exceeded.

Side-effect safety#

Simulation defaults to mocking all Tools and raising an error when no mock matches. Keep that default for production regression suites. Selecting no mocks or fallback to a real Tool can send an external request using the current live Secret value.

Tool Invocation assertions capture calls without performing HTTP. A Simulation explicitly allowed to call real Tools is handled conservatively during recovery: stale work becomes interrupted instead of being repeated automatically.

Use synthetic history and variables, never credentials or unnecessary customer data. Review Tool Invocation and mocks and Durable execution and recovery before enabling real integrations.