Test transcribers in the dashboard

Organization admins have transcriber testing in the OnCore dashboard already, with nothing to build. You flag a wrong line while you read a call, the scores fill in on their own, and a full-call run starts from any flagged line in a bank.

Overview#

The dashboard reads and writes the same flags, banks and runs as the Transcriber Tests API, through the same endpoints. A bank created here can be read over the API, and lines your own product flags show up in the dashboard's tables, so the two are safe to mix. To put these screens inside your own product instead, see Build a transcriber testing UI.

This page assumes you know what a bank, a control line and a full-call run are. If not, read the introduction first.

Prerequisites#

• An organization admin account. Members do not see transcriber testing in the sidebar, and their call pages have no way to flag a line.
• Recorded calls. The call page offers flagging only on a call with a recording, and turning recording on only covers calls from then on. See Recording and transcripts.
• One way of writing things. Scoring ignores capitals and punctuation but compares the words exactly, so agree how to write times, numbers and spellings before people start flagging. The introduction explains why "830" and "eight thirty" do not match.

Find transcriber tests#

Open Transcriber tests in the sidebar, just below Tests. It has three tabs:

• Results has one card per bank, with every transcriber's accuracy.
• Flagged turns lists every flagged line and control line in your organization, across all its tenants.
• Full calls lists the full-call runs of every bank, newest first.

Flagging itself happens on the call page, covered below. If the page says no organization is selected, select one in the sidebar.

Create a bank#

On Results, click New bank and fill in the dialog.

There is no field for the live transcriber. Each line's live transcriber is the one its call ran on, so one bank can hold lines from calls on different transcribers, and the bank shows each of them.

Click Create bank. The new card says No lines scored yet until its first line is scored.

To rename the bank or change its threshold or control lines later, open it from its card and click Edit bank. Candidates have their own dialog, described in Change a bank's candidates.

Flag a line on a call#

The panel closes and the line gets a badge saying where it has got to, such as Flagged, waiting for a worker or Flagged, scored in English. The banner counts the call's flags. Click Done to leave flagging.

A line flagged into no bank waits unscored until you assign it to one from Flagged turns.

A line carries one flag. If someone else flags it while you are writing, your text is not saved: the panel says This line is already flagged. and reopens showing their flag, which you can then edit. A control line sampled from an earlier flag can still be flagged; your flag replaces the control.

Change or remove a flag#

With flagging on, click a flagged line to reopen its panel with what was saved. Change the text, the tags, the note or the bank, then click Save. Moving a flag to another bank takes its control lines with it and scores them all against that bank's candidates.

Remove flag asks first. Removing takes the line, and the control lines sampled with it, out of the bank's scores.

Flagged turns can also edit what was said, the tags and the note, and put a line that is in no bank into one. Removing a flag, or moving it from one bank to another, happens on the call.

Follow the scoring#

Scoring starts as soon as a line is flagged into a bank, and runs in the background, so there is no run to start. The call page, the bank page and Flagged turns refresh on their own while any line they show is still being scored. Once you have clicked Load more on a table it stops refreshing, so reload the page to see new scores.

A line's status reads:

When your organization has too much scoring queued, a new flag into a bank is refused with a message saying so, and nothing is saved. Try again once some of it finishes, or flag the line into no bank for now.

Read the results#

Results shows one card per bank. Under the bank's name, a line lists the live transcribers its lines ran on and how many lines each has. A badge names the leader and gives the headline: Nova 3 clears 60% when the leader is at or above the threshold, Nobody clears 60% when it is not, and No lines scored yet before anything is scored.

Each candidate gets a bar with its accuracy. A candidate at or above the threshold is green, and the leader is highlighted while it is still under. The amber mark across every bar is the threshold. The rows marked (live) come last, in grey. Each shows a live transcriber's accuracy over its own lines. Read them for reference, not as rivals to the candidates; the introduction explains why.

The window picker sets how far back the scores look: Last 7 days, Last 30 days (the default) or Last 90 days. Export CSV downloads every card for the chosen window. Read the number of scored lines before trusting a gap between two bars; on ten lines, one miss moves accuracy by ten points.

Click a card to open the bank.

Dig into one bank#

The bank page shows its threshold and control lines per flag, the candidates, and the live transcribers on its lines. Below them, Turn by turn has one row per line from the last 30 days, whatever window Results is set to.

• Line gives the tenant and when the line was flagged, a control badge on control lines, and the line number with its tags. Flagged rows also carry Test on the full call.
• What the caller said is the text the flagger wrote. On a control line it is the live transcript.
• One column per live transcriber, marked live, then one per candidate. A tick is an exact match, and red text is what the transcriber wrote instead. Under either, WER is the word error rate, and a note such as case means the raw text differed only in capitals or punctuation. Failed means the transcriber returned an error; hover over it for the message. A dash in a live column means the line's call ran a different transcriber.
• judge pick marks the candidate the judge chose when several matched the words but wrote them differently, such as "Mary-Kate" and "mary kate". Hover over it for the judge's reason. The judge never changes a tick or a score.

The play button at the start of each row plays that line. Filter the table to Flagged or Controls lines, or to one tag, and Export CSV downloads the rows those filters select.

Change a bank's candidates#

Click Edit candidates. Tick the transcribers to add and untick the ones to drop, up to eight in all; newly ticked ones carry a new badge.

When you add a candidate, the dialog counts the bank's flagged and control lines from the last 30 days that the new candidates have not scored, and offers to score them, with the audio length and an estimated cost. That option is ticked by default. Untick it and the new candidate only scores lines flagged from then on. When your organization has too much scoring queued, Save and score is refused and nothing is saved; untick the offer to save the list on its own.

Ticking and unticking in the list saves nothing and spends nothing. The bank changes only when you click Save, or Save and score when the offer is ticked. A candidate you drop leaves the cards and the table; add it back later and its earlier results return without being run again.

Work through flagged lines#

Flagged turns lists every flagged line and control line in your organization, newest first. Filter it by bank (including Not in a bank), by kind (Flagged, Controls or both) and by status. The Scoring status also lists lines that are in no bank, because they wait unscored.

Each row has a play button, the call's tenant, date and line number, what the caller said, the tags, the Live transcriber the call ran on, the bank and the status. On a line in a bank, the text under What the caller said is a link: it opens the bank page with that row scrolled into view and highlighted.

A line in no bank shows Assign to a bank in place of the bank. Pick one and the line moves there and is scored, without control lines; the picker is disabled until your organization has a bank. Edit changes what was said, the tags and the note. A line in a bank, control lines included, also has Test on the full call.

Prove the leader on whole calls#

A full-call run replays one candidate on every caller line of up to ten calls, compares the result with what each call stored, and has the judge label every difference. Run one before you switch an assistant to the leader. The switch itself is a change you make on the assistant; see Providers.

Besides the candidate, up to 20 of the run's lines, the calibration lines, also go through the live transcriber again, to check that replaying the recording hears what the call heard. The estimate includes them. It covers transcription only, and the judge's analysis is billed on top, so the cost the finished run shows can come out a little higher.

Your organization runs one full-call run at a time. While another is queued or running, starting one is refused, with a link to Open the run in progress.

Read a full-call run#

While a run is going, its page shows Replaying caller lines with a count and a bar, then Writing the analysis while the judge works. The page refreshes on its own. You can leave it, and the run keeps going. Cancel run stops it after you confirm, and the judge then writes no verdict.

A finished run reads from the top down.

• The verdict is a short summary written by the judge from its labels. Check the sample before you act on it. A failed run shows its error above the verdict.
• Five tiles count the calls replayed, the caller lines, the lines that differ, the production mistakes fixed (lines the candidate got right and the live transcript got wrong) and the new mistakes (the reverse).
• Where they disagree splits the lines into identical and differing, then breaks the differing ones down by the judge's label: the candidate is right, same meaning with different writing, the live transcriber was right, or both wrong. Checked by a person gives the same breakdown for the lines someone has reviewed. The judge does not label lines that have a flag; those are decided by comparing with what the flagger wrote. Recording path agreement is the share of the calibration lines that came out the same when replayed. Below about 90%, part of the disagreement comes from the recording rather than the transcriber, so give the run's numbers less weight.
• What the candidate broke groups the new mistakes into patterns, each with an example of the live text and what the candidate wrote instead, and how many lines it covers.
• Call by call lists the run's calls with the most disagreement first, with each call's length, lines, differing lines and new mistakes.

Check the sample#

Check the sample shows one sampled line at a time, with a play button, what the judge said, and the live and candidate texts side by side. Listen, then click [candidate] is right, Live was right, Same meaning or Both wrong. The next line appears, and the counter shows how many of the sample are checked. Your answers feed Checked by a person, and they show how far to trust the judge's labels on the lines nobody checked.

Rescore, export or cancel#

Once a run has ended, its page offers Export CSV and Rescore; while it is going, Cancel run is the only action. Export CSV downloads every line of the run. Rescore recomputes every score from the transcripts the run kept, at no transcription cost; use it after correcting what was said on a flagged line, so the run picks up the correction. On a completed or failed run, the rescore dialog can also have the judge relabel the differing lines and write a new verdict, which costs a little for the judge. That is also the retry when a run's lines were transcribed but its analysis failed. A cancelled run can be rescored but not relabeled.

Limits in the dashboard#

A few things work differently here than over the API, or not at all.

• No retry for a failed line. Saving an unchanged candidate list only closes the dialog. Retry over the API by saving the same candidates with scoring turned on (see Change a bank's candidates), or remove the flag on the call and flag the line again.
• Some views lag. The Results cards, the bank page header and the bank lists in pickers can take up to a minute to show a change, and Results does not refresh on its own. Full calls shows the newest 50 runs of each bank and does not refresh on its own either.
• Sample answers are final here. Answering in Check the sample updates the tiles and Where they disagree only when you reload the page, and an answer cannot be changed in the dashboard.
• Control lines follow their flag. A control moves and is removed with the flag it was sampled for, and cannot be moved or removed on its own. On the bank page, only flagged rows offer Test on the full call.
• Line numbers count every entry. "line 7" is the line's position in the call's message list, counted from 0 and including assistant lines and tool calls.
• The remembered bank is per browser. The Bank default is kept per organization in each browser, and changing a flag's bank while editing it changes the default too.
• Flagging needs the call's lines. Flag for transcription does not appear on a call with no recording or no stored transcript, or when the dashboard cannot load the call's lines; reload the page to try again.
• Suggested tags only. The flag panel and the bank page's tag filter offer the suggested tags. Over the API, any tag of lowercase letters and underscores is accepted.
• One tenant per run. The dashboard adds calls from the flagged line's tenant. Over the API, a run can take any calls of your organization, and it does not need to start from a flagged line.
• Fixed windows and calibration. The dashboard offers windows of 7, 30 and 90 days, and checks up to 20 calibration lines on each run. The API takes other values within its limits.

The Transcriber Tests API reference has every endpoint and field.

Next steps#

• Transcriber testing: how banks, control lines, scoring and the judge work.
• Build a transcriber testing UI: the same screens in your own product, on the REST API.
• Recording and transcripts: turn on the recordings every test needs.