Universal-3.6 Pro

AssemblyAI's newest realtime model, universal-3-6-pro, and the one to choose for a new AssemblyAI transcriber. It judges the end of a turn from punctuation rather than a confidence score, and carries prior turns forward as context so a short reply like "seven pm" is read in light of the question that prompted it. It covers 32 languages, 13 more than Universal-3.5 Pro, and takes exactly the same settings.

Reached through the LiveKit Inference gateway.

A refused value does not end the call at once on this model. The session opens, is rejected, and is retried, and after about 20 seconds the call ends. For those 20 seconds nothing the caller says is transcribed, so the assistant never answers them.

Settings#

Every setting is optional. Leave one out and OnCore sends nothing for it, so the Default applies. If invalid is what the provider does with a value it does not accept.

Overriding on a single call#

Return this from your assistant-request webhook to change these settings for one call only. The stored transcriber is left untouched, and the next call starts from it again.

{
  "data": {
    "overrides": {
      "transcriber": {
        "settings": {
          "mode": "balanced",
          "prompt": "A caller booking a table at a restaurant.",
          "keytermsPrompt": [
            "Sadie",
            "reservation"
          ],
          "minTurnSilence": 200,
          "maxTurnSilence": 800,
          "vadThreshold": 0.5,
          "voiceFocus": "near-field",
          "voiceFocusThreshold": 0.5,
          "continuousPartials": true,
          "interruptionDelay": 300,
          "speakerLabels": false
        }
      }
    }
  }
}

Send only the keys you want to change: anything you leave out keeps the value the assistant already has. Add "id" beside "settings" to swap the whole transcriber for another one from your catalogue at the same time.

mode: Leaving mode out is not the same as balanced. On an eight-second reply of two sentences, the last transcript arrived about 0.4s after the caller stopped with mode unset, 0.9s on min_latency, 1.5s on balanced and 2.8s on max_accuracy. Unset also closed the turn at the first full stop, where each of the three presets held both sentences as one turn. The names are case-sensitive: BALANCED is refused, and so is an unknown name such as fast. An empty string is accepted and behaves as unset.

prompt: A prompt of 1800 characters was refused and the call failed. No length between a few sentences and 1800 characters was tested, so keep it short. Describing the call is what works: naming a restaurant and a surname in the prompt corrected both spellings on a recording the model otherwise got wrong. A formatting instruction does not work: asking for numbers spelled out as words changed nothing. prompt and keytermsPrompt work together in one body.

keytermsPrompt: 101 terms are refused and the call fails. Terms from a dictionary attached to the assistant are added to this list on every call, so keep the two together at 100 or fewer. AssemblyAI ignores a term longer than 50 characters rather than refusing it, and an empty list is accepted. Three terms corrected a made-up restaurant name and an unusual surname that the model misspelled without them.

minTurnSilence: Never refused: negative, fractional and very large values were all accepted. AssemblyAI describes it as the silence before the model checks for terminal punctuation, but on its own it had no measurable effect: 50 and 2000 gave the same turn timing, both on a reply with a full pause and on one followed by a long silence. To change when a turn ends, set maxTurnSilence.

maxTurnSilence: The strongest end-of-turn lever on this model, and raising it makes turns end later. AssemblyAI describes it as the silence after which a turn ends regardless of punctuation, but the model waits out most of it even after a finished sentence: on a reply followed by silence, 1500 delayed the last transcript by about 1.3s against leaving it unset, 3000 by about 2.8s and 5000 by about 4.8s. No value tested ended a turn sooner than leaving it unset. A negative or fractional value is refused and the call fails. 0 and very large values are accepted. It is also what holds a sentence together across a pause: at 3000, a sentence broken by a pause after "a table for" came back as one transcript instead of two.

vadThreshold: Never refused, and no measurable effect anywhere it was tested. Every value from 0.05 to 1.5 gave the same transcript on quiet speech and on speech over background noise, so a value above 1 does not silence the call. At 0, noise alone and silence alone still produced no transcript. Values outside 0 to 1, negative ones included, are accepted.

voiceFocus: Case-sensitive: NEAR-FIELD is refused, and so is mid-field. An empty string is accepted and behaves as unset. Neither value changed the transcript of a caller with another voice talking in the background.

voiceFocusThreshold: Refused unless voiceFocus is also set. Sent on its own, even a valid 0.5 fails the call, so a per-call override that adds only this key fails on any transcriber that does not already store voiceFocus. With voiceFocus set, 0 and 1 are accepted and 2 and -1 are refused.

continuousPartials: false roughly halves the interim results: 6 instead of 12 or 13 on an 18-second reply. The final transcript was the same.

interruptionDelay: A value above 1000, below 0, or with a fraction is refused and the call fails. AssemblyAI describes it as how soon the first partial arrives. Measured at 0 and at 1000, the first partial arrived at the same moment, about 1.5s into the audio, so do not rely on it to speed up barge-in.

speakerLabels: true adds a speaker id to each final transcript and splits speech into shorter finals: a two-voice exchange came back as eight finals instead of five. The ids are not one per person. Two voices came back as three ids.

The settings above are the complete set for this model. AssemblyAI's own documentation covers a wider API surface than a voice assistant uses, and a key that is not in the table has no effect here.

Unlike some providers, an unrecognised key here is tolerated rather than fatal, and so are the parameters belonging to AssemblyAI's Universal-Streaming models: they are accepted and ignored rather than refused.

Send the types in the table. Some wrong types are refused and fail the call, such as a number for mode or voiceFocus. Others are accepted, such as a quoted number for maxTurnSilence or interruptionDelay, and whether an accepted wrong type takes effect was not measured.

Turn detection#

AssemblyAI describes this model's turn detection as punctuation-based: after a short silence it looks for a terminal ., ? or ! to decide the turn has ended. Measured, maxTurnSilence and mode are the two settings that move the end of a turn, and both move it later than leaving them unset. minTurnSilence did not move it, on its own.

Even with no full stop, a pause in the middle of a sentence ended the turn at the defaults. Raise maxTurnSilence (3000 was measured) or set mode to max_accuracy, either one, and this model holds the sentence together as one turn. The two were not tested together. The cost is the wait: with either setting the turn closed about 3 seconds after the caller finished.

Languages#

Each transcriber in the catalogue carries its language: multi, or one of these 32 codes. A pinned entry names its language, as in AssemblyAI Universal-3.6 Pro (English), and the multilingual entry names none. Choose the entry from GET /transcribers and set it as the assistant's transcriberId, as shown in Pick a transcriber.

Use multi. It turns on AssemblyAI's per-turn language detection, labels each turn with the language it was spoken in, and never refuses a call. Its weak spot is speech in a language outside the table: Greek came back labelled Hebrew and written in Hebrew script, and Hungarian and Thai came back as garbled text under the wrong labels. No setting on this model transcribes those callers usefully, so give them a transcriber that covers their language.

A pinned language changes the label, not the words. Pinned to one code, the model still transcribes what the caller actually says and reports the pinned code as the language: a transcriber pinned to English transcribed German, French and Arabic callers in their own languages and labelled every turn en. So pinning does not keep a call to one language. Pinned to Dutch, a German caller came back in German with a single word in Dutch. Pin only when you want every turn reported as that one language.

A pinned code outside the table fails the call, and sooner than a refused setting does. The session is refused as the call connects, and the call ends within about 6 seconds with nothing transcribed. Greek (el), Hungarian (hu) and Thai (th) were each refused.

Moving from Universal-3.5 Pro#

Pick a Universal-3.6 Pro entry from GET /transcribers, pinned to the same language if the catalogue has one or multi otherwise, and set it as the assistant's transcriberId with PATCH /assistants/{id}. To try it on one call first, send its id in a per-call override. Each catalogue entry carries its own stored settings, so switching entries does not copy the old entry's; the settings you send in a per-call override apply to whichever entry the call runs on. Every setting was sent to both models with the same valid and invalid values, and each was accepted or refused identically, so settings that work on Universal-3.5 Pro or u3-rt-pro work here unchanged. Timing at the defaults matched too: with mode unset and on each of the three presets, the last transcript landed within a few hundredths of a second of the same point on both models. Transcriber settings in use on Universal-3.5 Pro in English, French, German, Italian, Spanish and English-French multilingual were each replayed on this model and transcribed normally.

Three things differ:

• 13 more languages: Afrikaans, Cantonese, Estonian, Galician, Korean, Marathi, Norwegian Nynorsk, Persian, Romanian, Russian, Urdu, Xhosa and Zulu. Korean, Romanian and Russian pins were measured failing the call on Universal-3.5 Pro and working here.
• Pauses: with a raised maxTurnSilence or max_accuracy, this model holds a sentence together across a mid-sentence pause, where Universal-3.5 Pro still splits it. See Turn detection.
• Pinned language: Universal-3.5 Pro can bend a transcript toward a pinned language, rewriting much of a German sentence into Dutch when pinned to nl. This model changed one word in the same test.

Next steps#

• Universal-3.5 Pro: the previous AssemblyAI model.
• Cartesia: speech-to-text from Cartesia.
• Providers introduction: per-call overrides, and how a wrong value behaves across providers.