Run text tests

A test suite belongs to one assistant. Cases live inside it. A run copies those cases at start time (suiteSnapshot), so later edits do not rewrite history.

Typical use: two cases against a bookings assistant, one that should get opening hours from a mocked tool, one that should be told they have the wrong number.

Overview#

Create the suite, add cases, start a run, poll GET /test-runs/{id} until it is no longer queued or running. The table below is the whole surface. Auth is the same as the rest of the API: Authorization: ApiKey YOUR_API_KEY, and X-Tenant-Id when you are on an organisation key. See Authentication.

Suite#

PATCH may send name, assistantId, or both. A name-only patch does not re-check assistant ownership. Changing assistantId does.

A thirteenth case is rejected. An empty suite cannot be run.

Creating a suite does not check the assistant's model, but running one does: the assistant must be on a chat model from GET /models. A suite for an assistant on a realtime (speech-to-speech) model saves fine and is refused at run time. See Run the suite.

Case#

Assertions#

A case passes only if the free-text judge passes and every required assertion passes. required defaults to true when omitted. An optional assertion that fails does not fail the case.

tool_called / tool_not_called

{ "type": "tool_called", "tool": "GetOperatingHours", "required": true }

extract

Pull a named value from the transcript and compare it.

{
  "type": "extract",
  "name": "offered_hours",
  "description": "Whether the assistant stated opening hours",
  "valueType": "boolean",
  "comparator": "eq",
  "expected": true
}

Create a suite and a case#

POST /test-suites, then POST /test-suites/{id}/cases.

curl -X POST https://core-api.heysadie.ai/test-suites \
  -H "Authorization: ApiKey YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Bookings smoke",
    "assistantId": "ASSISTANT_ID"
  }'

curl -X POST https://core-api.heysadie.ai/test-suites/SUITE_ID/cases \
  -H "Authorization: ApiKey YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Ask for Saturday hours",
    "personaInstructions": "You want to know if the shop is open this Saturday. Be brief.",
    "expectedOutcome": "The assistant states Saturday opening hours and does not transfer.",
    "maxTurns": 8,
    "toolMocks": {
      "GetOperatingHours": { "saturday": "09:00-17:00" }
    },
    "assertions": [
      { "type": "tool_called", "tool": "GetOperatingHours" },
      { "type": "tool_not_called", "tool": "TransferCall" }
    ]
  }'

Run the suite#

POST /test-suites/{id}/run

The body is optional. Send "mode": "text" or omit the body. "mode": "voice" returns 400.

The run talks to the assistant on its configured model and nothing else. There is no fallback to another model: if the configured model fails to answer, that case ends with status: "error" and the provider's message in its error field, rather than a result produced by a model you did not configure. One case in error puts the whole run in error, so a poller sees error, not failed.

If the assistant's model has type: "realtime" (a speech-to-speech model such as gpt-live-1), the request returns 400 with VALIDATION_ERROR and a message naming the model. Nothing is queued and nothing is billed. Those models have no text mode, so a text run cannot exercise them. Retarget the suite with PATCH /test-suites/{id} to an assistant on a chat model, or test the voice model on a real call.

iterations is optional. Integer from 1 to 10. Default 1. Each case runs that many times in the same run. When it is greater than 1, each case result includes passCount, runCount, and attempts. The suite still reports one row per case. The case passed only if every attempt passed.

curl -X POST https://core-api.heysadie.ai/test-suites/SUITE_ID/run \
  -H "Authorization: ApiKey YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"iterations": 3}'

The response is a run in queued. The run id is data.id. Poll GET /test-runs/{id} until data.status is passed, failed, or error. Two seconds is a reasonable client interval.

curl https://core-api.heysadie.ai/test-runs/RUN_ID \
  -H "Authorization: ApiKey YOUR_API_KEY"

data.status is one of:

Timeouts are applied when you GET the run: queued for more than 10 minutes, or running for more than 30, becomes error.

caseResults fills in as the run goes, so a client polling a running run can show progress instead of a blank wait. While the status is running, every case carries passCount and runCount whatever iterations is set to, and runCount is the number of attempts finished so far, not the target. Compare it against iterations to get a fraction. Transcripts are empty until the run finishes, so do not read them mid-run.

Once the status is passed, failed, or error, caseResults is final. Open a case in that object for the judge reason, assertion results, transcript, and mocked tool calls. passCount and runCount are then present only when iterations is greater than 1, as described above. costUsd is the priced sum of every case in that run. Usage totals Tests as the sum of costUsd for the same tenant and date window, not a single run.

Deleting the suite hides it and its cases. GET of that suite then 404s. The run stays, with suiteId still set and suiteSnapshot still holding the cases as they were.

List suites#

GET /test-suites

curl "https://core-api.heysadie.ai/test-suites?assistantId=ASSISTANT_ID" \
  -H "Authorization: ApiKey YOUR_API_KEY"

List runs#

GET /test-runs

curl "https://core-api.heysadie.ai/test-runs?suiteId=SUITE_ID" \
  -H "Authorization: ApiKey YOUR_API_KEY"

What a run does not do#

No phone call. No speech. No SMS, email, or transfer. No row in call history. Customer tool URLs are not hit; supply toolMocks for the JSON you want the assistant to see. Knowledge-base lookup still runs.

Next steps#

• Create and configure an assistant: the assistant a suite targets.
• Custom tools: the tools you mock by product name in toolMocks.

New to this? Testing explains what a suite is for, without the API detail.