Calls
In the browser
The browser half of live-call monitoring: connect to a call in progress, play its audio, read its transcript as it is spoken, and speak to the caller after a takeover.
Introduction is the overview. The monitoring API covers the six REST endpoints, and they are the whole of the feature except for two things: hearing the call, and speaking on it. Neither is a REST feature. The audio and the live transcript are a continuous connection to the call, so no endpoint can return them and there is no server-side transcript feed to poll.
This page is that second half. Read the REST page first; nothing here works without it,
because the credential this page connects with comes from
POST /calls/{id}/monitor-session.
Two ways to build it#
| What it is | When to pick it | |
|---|---|---|
@oncore/live-call-sdk | A small wrapper around livekit-client that hides the wire details below. Currently preview and not on a public registry | You want the shortest path, and you can wait for the package |
livekit-client directly | The public LiveKit browser client, plus the wire protocol documented below | You want to build today, or you already use livekit-client |
Both connect to the same call with the same credential and see the same messages. The SDK is
convenience, not a requirement: it is a few hundred lines over livekit-client, and
everything it knows is written down on this page.
The REST API is available, the browser package is preview
@oncore/live-call-sdk is not distributed on a public package registry, so the SDK examples
below cannot be installed from npm yet. The protocol they implement is documented in
The wire protocol and is stable, so a livekit-client integration
written against it today keeps working when the package ships.
Capabilities and availability is the
canonical status.
The wire protocol#
Everything a browser needs to know about a monitored call, in one place. This is what the SDK
implements; a livekit-client integration implements the same thing.
Connect with url and token from the monitor-session response. Then:
| Concern | Where it lives |
|---|---|
| The caller's and assistant's audio | Ordinary subscribed audio tracks. Attach them to one <audio> element |
| The live transcript | Text stream on topic lk.transcription, with lk.segment_id and lk.transcription_final attributes on each chunk. Not RoomEvent.TranscriptionReceived, which carries the same segments again |
| What was said before you joined | Send a hello on oncore.monitor.v1; the assistant replies with a history message |
| Who is talking to the caller now | control messages on oncore.monitor.v1, and the assistant's own lk.agent.state participant attribute |
| Server commands to the assistant | Topic oncore.control.v1. Authenticated with a per-call secret your browser never receives. Publishing here does nothing; ignore the topic |
The hello, published as JSON on oncore.monitor.v1 once connected:
{ "v": 1, "type": "hello" }The two replies, both JSON on the same topic, both carrying the protocol version:
{
"v": 1,
"type": "history",
"callId": "0195f1e2-3333-7000-8000-000000000000",
"control": "agent",
"messages": [
{ "role": "user", "text": "Hi, I need to move my appointment.", "at": 1770000000000 },
{ "role": "assistant", "text": "Of course. What day suits you?", "at": 1770000002500 }
]
}{
"v": 1,
"type": "control",
"callId": "0195f1e2-3333-7000-8000-000000000000",
"control": "human",
"at": 1770000045000
}control is one of agent, bridging, human, resuming, transferring, ending, or
unknown. Treat an unrecognised value as unknown rather than failing: the set can grow, and
a panel that throws on a new state is worse than one that shows a label it does not know.
Three things that will bite you
Read the text stream and nothing else. The assistant publishes every segment twice: once
on the lk.transcription text stream, and once through livekit-client's deprecated
RoomEvent.TranscriptionReceived. They are the same segments in two different shapes, so
listening to both renders the whole conversation twice. Register the text stream handler and
do not add that event listener.
Render segments by id, do not append them. Within the stream, the interim chunks and the
closing chunk of one utterance all carry the same lk.segment_id, and only the last has
lk.transcription_final true. Appending every chunk repeats a sentence as it is being
spoken; keying on the id replaces it in place.
The microphone is refused for a moment after takeover. livekit-client checks publish
permission against its own local copy of the grant, and that copy takes a moment to catch up
after POST /calls/{id}/takeover has changed it server-side, so the first
setMicrophoneEnabled(true) can be refused by a copy that has not caught up. If you are
building on livekit-client directly, this one is yours to handle: retry on a short bounded
budget, roughly a second, before treating it as a real failure. The SDK does it internally, and
Opening the microphone covers what that changes about the error
codes.
Nothing else on the connection is part of the contract. oncore.monitor.v1 is read-only with
respect to the call: the only message it accepts is the hello, and the only thing it returns is
the conversation you are already listening to.
Connect the browser#
@oncore/live-call-sdk declares livekit-client as a peer dependency with a permissive range, so a page using the SDK carries both, and it must carry exactly one copy of livekit-client. Two copies in one page is a subtle source of connection failures, and it is the first thing to check when a connection opens and then hears nothing.
connectMonitor takes the join material your session route passed through and returns a connection:
import { connectMonitor } from "@oncore/live-call-sdk";
const connection = await connectMonitor({
url: session.url,
token: session.token,
agentIdentity: session.agentIdentity,
});| Option | Required | Description |
|---|---|---|
url, token, agentIdentity | Yes | Straight from the monitor-session response, unmodified |
connectTimeoutMs | No | How long to wait for the connection. Default 15000 |
audioTimeoutMs | No | How long to wait for the first subscribed audio track. Default 15000 |
historyWaitMs | No | How long to wait for the conversation replay before resolving anyway. Default 2000, and 0 skips the wait |
micPermissionRetryMs | No | How long the SDK keeps retrying a microphone that takeover has just authorized. See Opening the microphone before changing it |
The promise resolves once you are connected, audio is subscribed, and the replay has either arrived or run out its short wait. So a panel usually paints complete on the first render, an assistant that never answers the replay costs a blink rather than an error, and the history event still covers a late arrival.
| Member | What it does |
|---|---|
history | The conversation so far. Usually populated by the time the promise resolves |
control | Who is talking to the caller: agent, bridging, human, resuming, transferring, ending, or unknown |
agentState | What the assistant itself is doing, or null until it first reports |
on('history' | 'transcript' | 'control' | 'agentState' | 'disconnected', cb) | Subscribe. Each call returns its own unsubscribe function |
attachAudio(element) | Route the room's audio into a media element. Returns a detach function |
setMicrophoneEnabled(enabled) | Open or close the supervisor's microphone. Rejects with a MonitorError when it cannot. Only succeeds after takeover |
disconnect() | Leave the room |
history and agentState are properties as well as events for the same reason: a component that attaches its listeners after the promise resolves would otherwise be blind until the next change. Read the property once on mount, then keep it current from the event.
A transcript line's role is user for the caller, assistant for the AI, or system. A streamed segment is only ever user or assistant. system arrives only in the replay, and covers the labelled non-conversational lines the platform injects, the takeover marker among them. A panel that is already attached learns about a takeover from the control event, never from a segment.
agentState is separate from control, and the two answer different questions: control is who is talking to the caller, agentState is what the assistant is doing. Its value is LiveKit's lk.agent.state passed through unchanged, which today means initializing, listening, thinking, or speaking. That set belongs to LiveKit rather than to OnCore, so render an unrecognised value rather than switching exhaustively on it. An unfamiliar control value arrives as unknown for the same reason.
One security property worth knowing, because it shows up as a silence rather than an error. Every monitor token may publish data packets, so a second panel in the same room could in principle send a forged control or history message. The SDK believes those messages only from agentIdentity and drops the rest, which is why you pass agentIdentity through from the session response rather than guessing it.
History and live segments overlap#
The transcript arrives on two paths and reconciling them is your job.
history is a snapshot: the assistant replays what has been said so far, because a live transcript has no backlog for someone who joins in the middle of a call. transcript then streams each new segment, first as an interim guess and again once it is final. The last message in history and the first streamed segment can be the same utterance, because the replay is assembled at the moment you join and speech does not stop to wait for it.
The two also carry different clocks. TranscriptMessage.at is when the assistant recorded the line, off the assistant's clock. TranscriptSegment.receivedAt is when the browser received the segment, off your user's clock. Both are epoch milliseconds, and a merged list must never be sorted by timestamp. Render the replay in the order it came in, then append streamed segments in arrival order. Sorting looks correct in testing, where the two clocks happen to agree, and scrambles the transcript on the one machine whose clock is off.
Only you know how you want a duplicate to look, so the SDK does not choose. The cheapest rule that works is to drop a replayed line when a final segment arrives carrying the same role and the same text:
import type { TranscriptMessage, TranscriptSegment } from "@oncore/live-call-sdk";
export type Line = { key: string; role: string; text: string; at: number; final: boolean };
export function fromHistory(messages: TranscriptMessage[]): Line[] {
return messages.map((message, index) => ({
key: `history-${index}-${message.at}`,
role: message.role,
text: message.text,
at: message.at,
final: true,
}));
}
export function mergeHistory(lines: Line[], messages: TranscriptMessage[]): Line[] {
// The replay can land after the first streamed segment, so it must not
// replace what is already on screen. Keep the streamed lines, drop any
// replayed line that duplicates one, and put the rest in front.
const streamed = lines.filter((line) => !line.key.startsWith("history-"));
const replayed = fromHistory(messages).filter(
(line) =>
!streamed.some(
(segment) => segment.final && segment.role === line.role && segment.text === line.text,
),
);
return [...replayed, ...streamed];
}
export function applySegment(lines: Line[], segment: TranscriptSegment): Line[] {
const next: Line = {
key: segment.id,
role: segment.role,
text: segment.text,
at: segment.receivedAt,
final: segment.isFinal,
};
// A final segment can repeat the tail of the replay. Drop the replayed
// copy and keep the streamed one, which carries a stable id.
const deduped = segment.isFinal
? lines.filter(
(line) =>
!(line.key.startsWith("history-") && line.role === next.role && line.text === next.text),
)
: lines;
const existing = deduped.findIndex((line) => line.key === next.key);
if (existing === -1) return [...deduped, next];
// An interim segment is replaced in place as the words firm up.
const merged = [...deduped];
merged[existing] = next;
return merged;
}Nothing here sorts, which is deliberate. Arrival order is the only order both paths agree on.
Opening the microphone#
The minted token grants subscribe only, so setMicrophoneEnabled(true) cannot work until POST /calls/{id}/takeover has returned. Take over first, then open the microphone. Nothing in the browser can lift that restriction, which is deliberate: a credential that reaches a browser can never make the assistant stand down on its own.
There is a race in the moment just after takeover, and the SDK absorbs it for you. livekit-client checks publish permission against its own local copy of the grant, and that copy takes a moment to catch up after takeover has changed it. So the first attempt immediately after a successful takeover can be refused by a copy that has not caught up, and that refusal looks identical to calling too early. The SDK retries internally on a bounded budget, micPermissionRetryMs, until the permission arrives or the budget runs out.
Do not add a retry of your own
Retrying setMicrophoneEnabled in your own code compounds with the SDK's, and makes a genuine failure take several times longer to reach the person waiting on it. Call it once and handle the rejection.
Because the SDK has already waited out the race, the two error codes now mean different kinds of thing:
MonitorError code | What it means | What to do |
|---|---|---|
mic_requires_takeover | The takeover genuinely has not happened | Fix the calling code: take over, wait for the response, then open the microphone |
mic_denied | The browser refused the permission, on the first attempt and with no retry | Explain the permission to the person and let them take over again |
mic_requires_takeover is therefore a programmer error to fix rather than a runtime condition to handle. The one honest cost of absorbing the race is that this particular mistake takes about a second to report instead of failing instantly. That second is what buys the race, and the race is the failure that would otherwise have shipped as a Take over button that works on a fast connection and not on a slow one.
Put the mint call behind your own route, and put the authorization check in it. getSession and user.permissions below are your own authentication and your own permission model; OnCore has no equivalent to call instead.
export const dynamic = "force-dynamic";
import { oncore } from "@/lib/oncore";
import { getSession } from "@/lib/auth";
export async function POST(
_request: Request,
{ params }: { params: Promise<{ id: string }> },
) {
const { id } = await params;
const user = await getSession();
// OnCore cannot make this decision for you. If this check is missing,
// every signed-in user can listen to every live call in the tenant.
if (!user || !user.permissions.includes("calls:monitor")) {
return Response.json({ error: "forbidden" }, { status: 403 });
}
const upstream = await oncore(user.tenantId, `/calls/${id}/monitor-session`, {
method: "POST",
body: JSON.stringify({ actorLabel: user.email }),
});
const body = await upstream.json();
if (!upstream.ok) return Response.json(body, { status: upstream.status });
// Pass the join material through untouched. The browser needs url, token,
// agentIdentity for the SDK, and monitorIdentity for the control calls.
return Response.json(body, { headers: { "Cache-Control": "no-store" } });
}oncore is one shared helper so no route can send a request in the wrong customer's context:
const BASE_URL = "https://core-api.heysadie.ai";
export function oncore(tenantId: string, path: string, init: RequestInit = {}) {
const apiKey = process.env.ONCORE_API_KEY;
if (!apiKey) throw new Error("ONCORE_API_KEY is not configured");
const headers = new Headers(init.headers);
headers.set("Authorization", `ApiKey ${apiKey}`);
headers.set("Content-Type", "application/json");
// Organization keys select the customer this way. A tenant key already
// carries its tenant, so pass an empty string and send no header.
if (tenantId) headers.set("X-Tenant-Id", tenantId);
return fetch(`${BASE_URL}${path}`, { ...init, headers, cache: "no-store" });
}The panel#
This component mints the session, connects, renders the transcript, and drives the three control calls contracted in the next section:
"use client";
import { useEffect, useRef, useState } from "react";
import { connectMonitor, MonitorError } from "@oncore/live-call-sdk";
import type { ConversationControlState, MonitorConnection } from "@oncore/live-call-sdk";
import { applySegment, fromHistory, mergeHistory, type Line } from "@/lib/transcript";
const CONFLICTS: Record<string, string> = {
not_live: "This call has already ended.",
agent_absent: "The assistant is reconnecting. Try again in a moment.",
not_monitorable: "This call cannot be monitored.",
monitor_cap_reached: "Too many people are already listening to this call.",
session_closed: "This session was closed. Open the call again to get a new one.",
held_by_other_monitor: "A colleague is already speaking on this call.",
};
// Close the monitor session when the panel goes away. keepalive lets the
// request outlive the page; navigator.sendBeacon cannot send a DELETE.
function closeSession(id: string, monitorIdentity: string | null) {
if (!monitorIdentity) return;
const query = new URLSearchParams({ monitorIdentity });
void fetch(`/api/live-calls/${id}/session?${query}`, {
method: "DELETE",
keepalive: true,
}).catch(() => {});
}
export function MonitorPanel({ id }: { id: string }) {
const connectionRef = useRef<MonitorConnection | null>(null);
const audioRef = useRef<HTMLAudioElement>(null);
const identityRef = useRef<string | null>(null);
const [control, setControl] = useState<ConversationControlState>("unknown");
const [lines, setLines] = useState<Line[]>([]);
const [micOn, setMicOn] = useState(false);
const [error, setError] = useState<string | null>(null);
useEffect(() => {
let cancelled = false;
const cleanups: Array<() => void> = [];
// Unmount is not the only way a panel goes away. pagehide covers the tab
// closing, which React never gets to hear about.
const onPageHide = () => closeSession(id, identityRef.current);
window.addEventListener("pagehide", onPageHide);
async function open() {
const response = await fetch(`/api/live-calls/${id}/session`, { method: "POST" });
const body = await response.json();
if (!response.ok) throw new Error(CONFLICTS[body.reason] ?? "Could not open this call.");
if (cancelled) return;
const connection = await connectMonitor({
url: body.url,
token: body.token,
agentIdentity: body.agentIdentity,
});
if (cancelled) {
await connection.disconnect();
return;
}
connectionRef.current = connection;
identityRef.current = body.monitorIdentity;
setControl(connection.control);
setLines(fromHistory(connection.history));
cleanups.push(
connection.on("history", (messages) =>
setLines((current) => mergeHistory(current, messages)),
),
);
cleanups.push(
connection.on("transcript", (segment) =>
setLines((current) => applySegment(current, segment)),
),
);
cleanups.push(connection.on("control", setControl));
cleanups.push(
connection.on("disconnected", (reason) => {
setMicOn(false);
setError(reason ?? "The call ended.");
}),
);
if (audioRef.current) cleanups.push(connection.attachAudio(audioRef.current));
}
open().catch((cause: unknown) => {
setError(cause instanceof Error ? cause.message : "Could not open this call.");
});
return () => {
cancelled = true;
window.removeEventListener("pagehide", onPageHide);
for (const cleanup of cleanups) cleanup();
void connectionRef.current?.disconnect();
connectionRef.current = null;
closeSession(id, identityRef.current);
};
}, [id]);
async function send(
action: "takeover" | "end",
extra: Record<string, string> = {},
attempt = 0,
): Promise<boolean> {
const monitorIdentity = identityRef.current;
if (!monitorIdentity) return false;
const response = await fetch(`/api/live-calls/${id}/${action}`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ monitorIdentity, ...extra }),
});
if (response.ok) return true;
const body = await response.json().catch(() => ({}));
// agent_absent is transient: the assistant is reconnecting and the call is
// still live. not_live is final and must never be retried.
if (body.reason === "agent_absent" && attempt < 2) {
await new Promise((resolve) => setTimeout(resolve, 750));
return send(action, extra, attempt + 1);
}
setError(CONFLICTS[body.reason] ?? `Could not ${action} this call.`);
return false;
}
async function takeOver() {
// Order matters. The token denies publishing until takeover has returned.
if (!(await send("takeover"))) return;
try {
// Once only. The SDK already retries the permission race internally.
await connectionRef.current?.setMicrophoneEnabled(true);
setMicOn(true);
} catch (cause) {
if (cause instanceof MonitorError && cause.code === "mic_denied") {
setError("Your browser blocked the microphone. Allow it, then take over again.");
} else {
// mic_requires_takeover reaching here means the takeover never landed.
// That is a bug in this component, not something the person can fix.
console.error("Opened the microphone before takeover completed", cause);
setError("Could not open your microphone.");
}
}
}
// There is no "hand back" call to make. Releasing the microphone and
// disconnecting is what returns the call: the assistant resumes on its own a
// few seconds after the supervisor leaves the room. Watch the `control`
// event for it rather than assuming the moment you disconnect.
async function stopSpeaking() {
await connectionRef.current?.setMicrophoneEnabled(false);
setMicOn(false);
}
const holding = control === "human" || control === "bridging";
return (
<div>
<p aria-live="polite">
{control === "bridging"
? "Handing the call to you"
: holding
? "You are speaking to the caller"
: `Assistant is handling the call (${control})`}
</p>
<ol>
{lines.map((line) => (
<li key={line.key} data-role={line.role} data-interim={!line.final}>
<b>{line.role}</b> {line.text}
</li>
))}
</ol>
{!holding && <button onClick={takeOver}>Take over</button>}
{holding && <button onClick={() => void stopSpeaking()}>Stop speaking</button>}
{holding && <button onClick={() => void send("end", { reason: "resolved" })}>End call</button>}
{micOn && <span>Microphone open</span>}
<p aria-live="polite">{error}</p>
<audio ref={audioRef} autoPlay />
</div>
);
}Render it as <MonitorPanel id={row.id} />. That is row.id, never row.callId.
The media element carries the room, so it plays the caller, the assistant, and a colleague who has taken the call over. Browsers block audio that starts without a user gesture. Opening this panel from a click on a call row usually supplies the gesture; if the media element still refuses to play, offer a button that calls element.play() and try again from there. Put the caller's audio behind a mute control of your own if your staff need one, by toggling muted on the same element.
Next steps#
- The monitoring API: the six REST endpoints this page's credential comes from.
- Browser calling: the same backend-mints-a-token pattern for placing a call from a page.
- Recording and transcripts: what the written and audio record contains afterwards.
- Errors and recovery: every refusal both halves can produce, in one table.