Durable execution and recovery

Agent Testing runs outside the request that creates an invocation. A worker claims bounded batches, performs model and evaluator work, records immutable evidence, and schedules remaining work. This design keeps large suites durable without holding a browser request open.

Claim ownership safely#

Each claimed run has a lease, a periodic heartbeat, and a fencing token. The lease marks temporary ownership, the heartbeat proves the worker is still alive, and fencing prevents an older worker from finalizing after ownership has moved. Slow model I/O does not hold a database transaction open.

A worker processes a bounded batch and re-enqueues the invocation when pending work remains. Duplicate jobs are safe because a run can be finalized only by the current lease holder. One executor failure does not cancel unrelated sibling runs.

Recover stale work#

Recovery considers only genuinely stale leases. A mocked run can return to pending for another safe attempt. A run allowed to call a real Tool becomes interrupted instead, because repeating it could duplicate an external side effect. Losing the lease or heartbeat cancels the old executor.

Quota reached during worker execution is stored as the run's failure_code; it is not returned as an HTTP 429 after the invocation already exists. The invocation summary is refreshed as terminal results arrive, without double-counting usage.

Recover results and bucketing#

Resubmit creates a new invocation and preserves the original. Use the old snapshot for reproduction or explicitly select current Draft or Live for comparison. Failure bucketing has its own claim and heartbeat. Retry is available only after every run is terminal and the earlier bucketing attempt failed.

Raw results survive malformed or failed bucketing output. Read Past runs and results before deciding to resubmit, and use Troubleshooting when retry state is rejected.