Introduction

OnCore can put a scripted caller in front of an assistant as text. You get a pass or fail and a transcript, and nobody has to dial the number.

What it is#

A test suite is a folder of cases aimed at one assistant. Each case is one caller (who they are, what they want) and one pass bar (what should happen). When you run the suite, OnCore talks to the real assistant in text, one case after another, each with a fresh session. OnCore scores the transcript against the outcome you wrote. Optional checks can require that a tool was called, that one was not, or that a fact extracted from the transcript matches a value you set. The suite passes only if every case passes.

Nothing leaves the platform. There is no phone leg, no speech-to-text, no text-to-speech, no SMS, no email, no transfer, and no row in call history. Tools the assistant would normally call on your servers are mocked, except knowledge-base lookup, which still runs against the assistant's documents. Token use for the run is billed on its own Tests line, separate from Calls.

What you can do with it#

• Prove a booking path without a live caller: a persona who wants a Tuesday slot, a mock for the hours tool, and a check that the assistant did not transfer.
• Catch a wrong-number greeting: seed the first turns so the assistant already thinks this is a misdial, then judge that it explains itself instead of taking a booking.
• Keep the same cases as the prompt changes: the suite belongs to the tenant and names an assistant. Change the assistant's instructions, run the folder again, compare.
• Read why a case failed: each finished run keeps a per-case table and the transcript for the case you open, including mocked tool calls.

How to set it up#

Create a suite for an assistant, add up to twelve cases, then start a run. The REST API is the setup path: Run text tests has the create, case, and run calls. The OnCore dashboard has the same folder under Tests (Suites to edit, Runs to launch), which is useful if people on the tenant prefer a screen to a script. Either way you are defining the same objects.

Limits & FAQ#

• v1 is text only. A voice run is rejected.
• A run talks to the assistant on the model the assistant is configured with, and nothing else. There is no fallback to another model, so a model that cannot answer shows up as an error on the case instead of a pass earned by a different model.
• The assistant must be on a chat model. A suite whose assistant runs a speech-to-speech model such as GPT-Live cannot be run as text: the run is refused with a message naming the model, in the API and in the dashboard alike. Switch the assistant to a chat model to test it in text, or test the voice model with a real call.
• A suite holds at most 12 cases. A case may run at most 40 caller turns (default 12).
• You cannot run a single case on its own. The run always walks the whole folder in the order you set.
• Knowledge-base lookup still hits the assistant's documents. Every other tool is short-circuited; you can supply the JSON it should return.
• Deleting a suite hides it and its cases. Runs already taken keep their snapshot and results.
• A run that sits queued for more than 10 minutes, or running for more than 30, is marked as an error the next time you fetch it.

Does this place a real call? No. There is no PSTN or WebRTC caller, and nothing is written to call history.

Will my CRM or SMS fire? No. Tool calls that would leave the platform are mocked. Knowledge lookup is the exception.

Which model answers in a test? The assistant's configured model, exactly as a caller would get it. Nothing is swapped in if that model fails, so a passing run is a pass for the model you actually ship.

Can I test a GPT-Live assistant? Not as text. Speech-to-speech models have no text mode, so the run is refused with a message that names the model. Point the suite at an assistant on a chat model, or verify the voice model on a real call.

Who can use it? Anyone on the tenant who can use assistants: member, organisation admin, and platform admin.

Does it cost? Yes. Each run's token use is priced and shown as Tests on usage, not folded into Calls.

Next steps#

• Run text tests: create a suite, add cases, start a run, and poll for the result.
• Create and configure an assistant: the assistant a suite targets.