Introduction

A caller says "Siobhan, table for four at eight thirty" and the transcript reads "Shivon, table for for at 830". With transcriber testing you mark that line, type what the caller actually said, and OnCore replays the line's audio through the transcribers you are considering and scores each one against your text. Over a few weeks of flags you see which transcriber gets your callers right, and before anyone changes an assistant you can replay the leader over whole calls and read what it would have fixed and what it would have broken.

What it is#

Transcriber testing is an organization-level quality loop for speech-to-text. It works on calls that already happened: nothing changes on a live call, and no assistant setting moves until you change it yourself.

There are two ways to use it, and they share the same flags, banks and runs. Organization admins use it in the OnCore dashboard, flagging lines from the call page and reading results under Transcriber tests; Test transcribers in the dashboard walks through it. Or you build the screens into your own product on the REST API, and Build a transcriber testing UI covers every screen and the requests behind it.

How you use it#

1. Record calls. Only recorded calls can be tested, and turning recording on only covers calls from then on.
2. Create a bank of the transcribers you want to compare, for example "English" with three candidates.
3. Flag lines as you find them. Whoever reads transcripts marks a wrong caller line, types what was said, and picks the bank. Scoring starts straight away.
4. Read the results. Each bank shows every transcriber's accuracy on your flagged lines and which one leads.
5. Run the leader on whole calls before switching, and read what it fixes, what it breaks, and a written verdict.
6. Switch the assistant yourself if the evidence holds. Transcriber testing never changes an assistant.

A flag can also be made outside any bank, for example while you decide where it belongs. It waits unscored, and is scored as soon as someone moves it into a bank.

The ideas behind it#

The live transcriber is the transcriber a call actually ran on. OnCore records it on the call (for older calls, it is found by name), so editing an assistant afterwards never moves a past line under a different transcriber. Its stored text is the live transcript, which the API calls production. The live transcript is scored as it was stored, never transcribed again, because it is the text the assistant really worked from. A call whose live transcriber is unknown cannot be tested, because there is nothing to compare against.

A flag is one caller line plus what was actually said, written word for word. You can add error tags (name, time, spelling, or your own) and a note. Each line can carry one flag.

Candidates are transcribers from the catalogue you want to compare. Each candidate transcribes the flagged line's audio, cut from the call recording with a small margin on each side, and its text is scored the same way the live transcript is.

A bank is a named set of candidates, such as "English" or "English + French", shared by everyone in your organization. A flag joins the bank you choose when you flag it, and its candidates are queued on it straight away. A bank also holds an accuracy threshold (60% unless you set another) and how many control lines each flag brings with it.

Control lines are caller lines from the same call that nobody flagged, sampled at random when you flag. Their expected text is the live transcript. They exist because a hard line is not a fair test on its own: a candidate that fixes the names your current transcriber misses but garbles ordinary sentences would win on flags alone. Controls catch that. A bank asks for zero, one or two per flag (one by default), and a call with fewer other caller lines gives fewer. The setting changes every score in the bank (more controls means more easy lines in the mix), so compare banks only when they use the same setting.

Scoring is mechanical, not an opinion. Both texts are lowercased, punctuation becomes a space (so "twenty-one" equals "twenty one", while apostrophes inside a word such as "don't" stay), and whitespace is collapsed. A match means the two normalized texts are equal. WER, word error rate, counts the words inserted, deleted or substituted against your text, divided by the number of words you typed. Digits and number words stay different on purpose: "830" does not match "eight thirty", because the systems reading a transcript treat them differently. Agree on one convention before people start flagging, such as writing times and numbers the way your booking system expects them, or two people will score the same line differently. A match whose raw text differed only in capitals or punctuation carries a note saying so.

Accuracy is matches divided by scored lines, per transcriber, over a window of days (30 unless you ask for another). A line where a transcriber failed to produce any text is counted as an error and left out of the division, so a provider outage does not read as bad transcription. Read the number of scored lines next to accuracy: on ten lines, one miss moves accuracy by ten points.

The live rows on a bank show, for each transcriber the bank's calls actually ran on, its accuracy over its own lines, for reference. They are not fair rivals to the candidates. Flags are mostly lines the live transcriber got wrong, and control lines are its own text, so a live row's score mostly reflects the mix of flags and controls.

The leader is the candidate with the highest accuracy. Ties go to the lower mean WER, then to the candidate listed first. It clears the threshold when its accuracy is at or above the bank's threshold. The threshold is a marker for your team; nothing happens automatically when a candidate crosses it.

The judge is a language model, and it never changes a score. On a flagged or control line where two or more candidates match but write it differently ("Mary-Kate" and "mary kate"), it picks the version that fits the business best, using the assistant's instructions and the neighbouring lines, and records one sentence of reasoning. On full-call runs it labels disagreements, as below.

A full-call run is the proof step before a switch. It usually starts from a call you flagged a line on, and you can add other calls, ten in total at most. The candidate you choose (the bank's leader unless you pick another) transcribes every caller line of those calls that has audio timing; the assistant's own lines are never included. Each line's result is compared with the live transcript. Where a line has a flag it is scored, and labeled, against your expected text; everywhere else the judge labels each line that differs: the candidate was right, the live transcript was right, they mean the same thing, or both are wrong. It then groups the lines the live transcript got right and the candidate got wrong into patterns ("drops surnames", "writes times as words") and writes a short verdict. A sample of the labeled lines, up to 120, is set aside for a person to check, so you can measure how far to trust the labels.

Calibration answers a question every full-call run raises: does transcribing a recording reproduce what the live call heard? A run also runs the live transcriber itself over the recording on a random sample of lines (20 by default), and compares the result with the stored live transcript. The share that come out the same is the run's audio path agreement. When it is low, differences between the candidate and the live transcript partly reflect the recording rather than the transcriber, and the run's numbers deserve less weight.

What you can do with it#

• Collect evidence while you work. A supervisor reading a call transcript flags the line that was wrong and types the correction. Scoring is queued the moment they flag, with no batch to wait for.
• Compare transcribers on your own callers. Put the candidates you are considering in one bank and watch accuracy build up over real names, menu items, addresses and accents, not a vendor benchmark.
• See exactly where each one fails. The turn-by-turn table shows every flagged line with each transcriber's text, a tick where it matched, and its WER where it did not.
• Prove a switch before making it. Run the leader over whole calls and read what it fixes, what it breaks, and a verdict, with a review sample to check the judge.
• Hand the numbers to someone else. The results cards, the turn table and every run's lines export as CSV.

Limits & FAQ#

• One organization, every tenant. Flags, banks and runs belong to your organization and span all its tenants, so everyone with access sees lines from every tenant's calls. In the dashboard that means organization admins; members do not see it. The API takes an organization key only; tenant keys cannot use it.
• Calls need a recording, and lines need audio timing. A line can be flagged only when the call was recorded, its live transcriber is known, and the line carries the start and end of the caller's speech. Recording is a per-assistant setting; see Recording and transcripts.
• Caller lines only. Assistant lines cannot be flagged, and full-call runs never include them.
• Eight candidates per bank, ten calls per run, one run at a time. A second run cannot start while another of your organization's runs is queued or running.
• Work is queued. Scoring runs in the background, a few jobs per organization at a time, so a flag can wait before it is scored. When too much is waiting, new flags and backfills that would add work are refused until the queue drains.
• You see the cost before large spends. Adding candidates to a bank and starting a full-call run both come with an estimate. The live transcript costs nothing; each candidate transcription is billed as speech-to-text audio, and the judge's work costs language-model tokens only.
• Runs are kept. A bank or a flag can be deleted. A full-call run cannot; you can cancel one that is still going. Runs are listed per bank, so deleting a bank takes its runs out of every list: keep the ids of runs you may want to read again.
• Playback is both channels. A line plays from the call recording as it was recorded, so you may hear the assistant underneath the caller.

Does flagging change the assistant or the call? No. Flags, banks and runs only read past calls. Switching an assistant to the winning transcriber is a separate change you make on the assistant; see Providers.

Which languages does it support? Any transcriber in the catalogue can be a candidate. A bank's language label is a display name only, so group candidates however suits your callers.

How many flags before we trust the leader? There is no fixed number. Watch the scored count on each bar, and confirm with a full-call run before switching, since that tests ordinary lines as well as the hard ones.

What if nobody flags anything in a bank for a while? The window only counts recent flags, so a bank's results show fewer lines and the leader can change. Widen the window (up to 90 days) to look further back.

Next steps#

• Test transcribers in the dashboard: flag lines, read the results and run full calls without writing code.
• Build a transcriber testing UI: every screen, the requests behind it, and what to do with each error.
• Transcriber Tests API: the generated reference for every endpoint.
• Recording and transcripts: turn on the recordings transcriber testing needs.