Skip to content

Calls

In the browser

The browser half of live-call monitoring: connect to a call in progress, play its audio, read its transcript as it is spoken, and speak to the caller after a takeover.

Introduction is the overview. The monitoring API covers the six REST endpoints, and they are the whole of the feature except for two things: hearing the call, and speaking on it. Neither is a REST feature. The audio and the live transcript are a continuous connection to the call, so no endpoint can return them and there is no server-side transcript feed to poll.

This page is that second half. Read the REST page first; nothing here works without it, because the credential this page connects with comes from POST /calls/{id}/monitor-session.

Two ways to build it#

What it isWhen to pick it
@oncore/live-call-sdkA small wrapper around livekit-client that hides the wire details below. Currently preview and not on a public registryYou want the shortest path, and you can wait for the package
livekit-client directlyThe public LiveKit browser client, plus the wire protocol documented belowYou want to build today, or you already use livekit-client

Both connect to the same call with the same credential and see the same messages. The SDK is convenience, not a requirement: it is a few hundred lines over livekit-client, and everything it knows is written down on this page.

The REST API is available, the browser package is preview

@oncore/live-call-sdk is not distributed on a public package registry, so the SDK examples below cannot be installed from npm yet. The protocol they implement is documented in The wire protocol and is stable, so a livekit-client integration written against it today keeps working when the package ships. Capabilities and availability is the canonical status.

The wire protocol#

Everything a browser needs to know about a monitored call, in one place. This is what the SDK implements; a livekit-client integration implements the same thing.

Connect with url and token from the monitor-session response. Then:

ConcernWhere it lives
The caller's and assistant's audioOrdinary subscribed audio tracks. Attach them to one <audio> element
The live transcriptText stream on topic lk.transcription, with lk.segment_id and lk.transcription_final attributes on each chunk. Not RoomEvent.TranscriptionReceived, which carries the same segments again
What was said before you joinedSend a hello on oncore.monitor.v1; the assistant replies with a history message
Who is talking to the caller nowcontrol messages on oncore.monitor.v1, and the assistant's own lk.agent.state participant attribute
Server commands to the assistantTopic oncore.control.v1. Authenticated with a per-call secret your browser never receives. Publishing here does nothing; ignore the topic

The hello, published as JSON on oncore.monitor.v1 once connected:

json
{ "v": 1, "type": "hello" }

The two replies, both JSON on the same topic, both carrying the protocol version:

json
{
  "v": 1,
  "type": "history",
  "callId": "0195f1e2-3333-7000-8000-000000000000",
  "control": "agent",
  "messages": [
    { "role": "user", "text": "Hi, I need to move my appointment.", "at": 1770000000000 },
    { "role": "assistant", "text": "Of course. What day suits you?", "at": 1770000002500 }
  ]
}
json
{
  "v": 1,
  "type": "control",
  "callId": "0195f1e2-3333-7000-8000-000000000000",
  "control": "human",
  "at": 1770000045000
}

control is one of agent, bridging, human, resuming, transferring, ending, or unknown. Treat an unrecognised value as unknown rather than failing: the set can grow, and a panel that throws on a new state is worse than one that shows a label it does not know.

Three things that will bite you

Read the text stream and nothing else. The assistant publishes every segment twice: once on the lk.transcription text stream, and once through livekit-client's deprecated RoomEvent.TranscriptionReceived. They are the same segments in two different shapes, so listening to both renders the whole conversation twice. Register the text stream handler and do not add that event listener.

Render segments by id, do not append them. Within the stream, the interim chunks and the closing chunk of one utterance all carry the same lk.segment_id, and only the last has lk.transcription_final true. Appending every chunk repeats a sentence as it is being spoken; keying on the id replaces it in place.

The microphone is refused for a moment after takeover. livekit-client checks publish permission against its own local copy of the grant, and that copy takes a moment to catch up after POST /calls/{id}/takeover has changed it server-side, so the first setMicrophoneEnabled(true) can be refused by a copy that has not caught up. If you are building on livekit-client directly, this one is yours to handle: retry on a short bounded budget, roughly a second, before treating it as a real failure. The SDK does it internally, and Opening the microphone covers what that changes about the error codes.

Nothing else on the connection is part of the contract. oncore.monitor.v1 is read-only with respect to the call: the only message it accepts is the hello, and the only thing it returns is the conversation you are already listening to.

Connect the browser#

@oncore/live-call-sdk declares livekit-client as a peer dependency with a permissive range, so a page using the SDK carries both, and it must carry exactly one copy of livekit-client. Two copies in one page is a subtle source of connection failures, and it is the first thing to check when a connection opens and then hears nothing.

connectMonitor takes the join material your session route passed through and returns a connection:

typescript
import { connectMonitor } from "@oncore/live-call-sdk";

const connection = await connectMonitor({
  url: session.url,
  token: session.token,
  agentIdentity: session.agentIdentity,
});
OptionRequiredDescription
url, token, agentIdentityYesStraight from the monitor-session response, unmodified
connectTimeoutMsNoHow long to wait for the connection. Default 15000
audioTimeoutMsNoHow long to wait for the first subscribed audio track. Default 15000
historyWaitMsNoHow long to wait for the conversation replay before resolving anyway. Default 2000, and 0 skips the wait
micPermissionRetryMsNoHow long the SDK keeps retrying a microphone that takeover has just authorized. See Opening the microphone before changing it

The promise resolves once you are connected, audio is subscribed, and the replay has either arrived or run out its short wait. So a panel usually paints complete on the first render, an assistant that never answers the replay costs a blink rather than an error, and the history event still covers a late arrival.

MemberWhat it does
historyThe conversation so far. Usually populated by the time the promise resolves
controlWho is talking to the caller: agent, bridging, human, resuming, transferring, ending, or unknown
agentStateWhat the assistant itself is doing, or null until it first reports
on('history' | 'transcript' | 'control' | 'agentState' | 'disconnected', cb)Subscribe. Each call returns its own unsubscribe function
attachAudio(element)Route the room's audio into a media element. Returns a detach function
setMicrophoneEnabled(enabled)Open or close the supervisor's microphone. Rejects with a MonitorError when it cannot. Only succeeds after takeover
disconnect()Leave the room

history and agentState are properties as well as events for the same reason: a component that attaches its listeners after the promise resolves would otherwise be blind until the next change. Read the property once on mount, then keep it current from the event.

A transcript line's role is user for the caller, assistant for the AI, or system. A streamed segment is only ever user or assistant. system arrives only in the replay, and covers the labelled non-conversational lines the platform injects, the takeover marker among them. A panel that is already attached learns about a takeover from the control event, never from a segment.

agentState is separate from control, and the two answer different questions: control is who is talking to the caller, agentState is what the assistant is doing. Its value is LiveKit's lk.agent.state passed through unchanged, which today means initializing, listening, thinking, or speaking. That set belongs to LiveKit rather than to OnCore, so render an unrecognised value rather than switching exhaustively on it. An unfamiliar control value arrives as unknown for the same reason.

One security property worth knowing, because it shows up as a silence rather than an error. Every monitor token may publish data packets, so a second panel in the same room could in principle send a forged control or history message. The SDK believes those messages only from agentIdentity and drops the rest, which is why you pass agentIdentity through from the session response rather than guessing it.

History and live segments overlap#

The transcript arrives on two paths and reconciling them is your job.

history is a snapshot: the assistant replays what has been said so far, because a live transcript has no backlog for someone who joins in the middle of a call. transcript then streams each new segment, first as an interim guess and again once it is final. The last message in history and the first streamed segment can be the same utterance, because the replay is assembled at the moment you join and speech does not stop to wait for it.

The two also carry different clocks. TranscriptMessage.at is when the assistant recorded the line, off the assistant's clock. TranscriptSegment.receivedAt is when the browser received the segment, off your user's clock. Both are epoch milliseconds, and a merged list must never be sorted by timestamp. Render the replay in the order it came in, then append streamed segments in arrival order. Sorting looks correct in testing, where the two clocks happen to agree, and scrambles the transcript on the one machine whose clock is off.

Only you know how you want a duplicate to look, so the SDK does not choose. The cheapest rule that works is to drop a replayed line when a final segment arrives carrying the same role and the same text:

lib/transcript.ts
import type { TranscriptMessage, TranscriptSegment } from "@oncore/live-call-sdk";

export type Line = { key: string; role: string; text: string; at: number; final: boolean };

export function fromHistory(messages: TranscriptMessage[]): Line[] {
  return messages.map((message, index) => ({
    key: `history-${index}-${message.at}`,
    role: message.role,
    text: message.text,
    at: message.at,
    final: true,
  }));
}

export function mergeHistory(lines: Line[], messages: TranscriptMessage[]): Line[] {
  // The replay can land after the first streamed segment, so it must not
  // replace what is already on screen. Keep the streamed lines, drop any
  // replayed line that duplicates one, and put the rest in front.
  const streamed = lines.filter((line) => !line.key.startsWith("history-"));
  const replayed = fromHistory(messages).filter(
    (line) =>
      !streamed.some(
        (segment) => segment.final && segment.role === line.role && segment.text === line.text,
      ),
  );
  return [...replayed, ...streamed];
}

export function applySegment(lines: Line[], segment: TranscriptSegment): Line[] {
  const next: Line = {
    key: segment.id,
    role: segment.role,
    text: segment.text,
    at: segment.receivedAt,
    final: segment.isFinal,
  };

  // A final segment can repeat the tail of the replay. Drop the replayed
  // copy and keep the streamed one, which carries a stable id.
  const deduped = segment.isFinal
    ? lines.filter(
        (line) =>
          !(line.key.startsWith("history-") && line.role === next.role && line.text === next.text),
      )
    : lines;

  const existing = deduped.findIndex((line) => line.key === next.key);
  if (existing === -1) return [...deduped, next];

  // An interim segment is replaced in place as the words firm up.
  const merged = [...deduped];
  merged[existing] = next;
  return merged;
}

Nothing here sorts, which is deliberate. Arrival order is the only order both paths agree on.

Opening the microphone#

The minted token grants subscribe only, so setMicrophoneEnabled(true) cannot work until POST /calls/{id}/takeover has returned. Take over first, then open the microphone. Nothing in the browser can lift that restriction, which is deliberate: a credential that reaches a browser can never make the assistant stand down on its own.

There is a race in the moment just after takeover, and the SDK absorbs it for you. livekit-client checks publish permission against its own local copy of the grant, and that copy takes a moment to catch up after takeover has changed it. So the first attempt immediately after a successful takeover can be refused by a copy that has not caught up, and that refusal looks identical to calling too early. The SDK retries internally on a bounded budget, micPermissionRetryMs, until the permission arrives or the budget runs out.

Do not add a retry of your own

Retrying setMicrophoneEnabled in your own code compounds with the SDK's, and makes a genuine failure take several times longer to reach the person waiting on it. Call it once and handle the rejection.

Because the SDK has already waited out the race, the two error codes now mean different kinds of thing:

MonitorError codeWhat it meansWhat to do
mic_requires_takeoverThe takeover genuinely has not happenedFix the calling code: take over, wait for the response, then open the microphone
mic_deniedThe browser refused the permission, on the first attempt and with no retryExplain the permission to the person and let them take over again

mic_requires_takeover is therefore a programmer error to fix rather than a runtime condition to handle. The one honest cost of absorbing the race is that this particular mistake takes about a second to report instead of failing instantly. That second is what buys the race, and the race is the failure that would otherwise have shipped as a Take over button that works on a fast connection and not on a slow one.

Put the mint call behind your own route, and put the authorization check in it. getSession and user.permissions below are your own authentication and your own permission model; OnCore has no equivalent to call instead.

app/api/live-calls/[id]/session/route.ts
export const dynamic = "force-dynamic";

import { oncore } from "@/lib/oncore";
import { getSession } from "@/lib/auth";

export async function POST(
  _request: Request,
  { params }: { params: Promise<{ id: string }> },
) {
  const { id } = await params;
  const user = await getSession();

  // OnCore cannot make this decision for you. If this check is missing,
  // every signed-in user can listen to every live call in the tenant.
  if (!user || !user.permissions.includes("calls:monitor")) {
    return Response.json({ error: "forbidden" }, { status: 403 });
  }

  const upstream = await oncore(user.tenantId, `/calls/${id}/monitor-session`, {
    method: "POST",
    body: JSON.stringify({ actorLabel: user.email }),
  });

  const body = await upstream.json();
  if (!upstream.ok) return Response.json(body, { status: upstream.status });

  // Pass the join material through untouched. The browser needs url, token,
  // agentIdentity for the SDK, and monitorIdentity for the control calls.
  return Response.json(body, { headers: { "Cache-Control": "no-store" } });
}

oncore is one shared helper so no route can send a request in the wrong customer's context:

lib/oncore.ts
const BASE_URL = "https://core-api.heysadie.ai";

export function oncore(tenantId: string, path: string, init: RequestInit = {}) {
  const apiKey = process.env.ONCORE_API_KEY;
  if (!apiKey) throw new Error("ONCORE_API_KEY is not configured");

  const headers = new Headers(init.headers);
  headers.set("Authorization", `ApiKey ${apiKey}`);
  headers.set("Content-Type", "application/json");
  // Organization keys select the customer this way. A tenant key already
  // carries its tenant, so pass an empty string and send no header.
  if (tenantId) headers.set("X-Tenant-Id", tenantId);

  return fetch(`${BASE_URL}${path}`, { ...init, headers, cache: "no-store" });
}

The panel#

This component mints the session, connects, renders the transcript, and drives the three control calls contracted in the next section:

components/monitor-panel.tsx
"use client";

import { useEffect, useRef, useState } from "react";
import { connectMonitor, MonitorError } from "@oncore/live-call-sdk";
import type { ConversationControlState, MonitorConnection } from "@oncore/live-call-sdk";
import { applySegment, fromHistory, mergeHistory, type Line } from "@/lib/transcript";

const CONFLICTS: Record<string, string> = {
  not_live: "This call has already ended.",
  agent_absent: "The assistant is reconnecting. Try again in a moment.",
  not_monitorable: "This call cannot be monitored.",
  monitor_cap_reached: "Too many people are already listening to this call.",
  session_closed: "This session was closed. Open the call again to get a new one.",
  held_by_other_monitor: "A colleague is already speaking on this call.",
};

// Close the monitor session when the panel goes away. keepalive lets the
// request outlive the page; navigator.sendBeacon cannot send a DELETE.
function closeSession(id: string, monitorIdentity: string | null) {
  if (!monitorIdentity) return;
  const query = new URLSearchParams({ monitorIdentity });
  void fetch(`/api/live-calls/${id}/session?${query}`, {
    method: "DELETE",
    keepalive: true,
  }).catch(() => {});
}

export function MonitorPanel({ id }: { id: string }) {
  const connectionRef = useRef<MonitorConnection | null>(null);
  const audioRef = useRef<HTMLAudioElement>(null);
  const identityRef = useRef<string | null>(null);
  const [control, setControl] = useState<ConversationControlState>("unknown");
  const [lines, setLines] = useState<Line[]>([]);
  const [micOn, setMicOn] = useState(false);
  const [error, setError] = useState<string | null>(null);

  useEffect(() => {
    let cancelled = false;
    const cleanups: Array<() => void> = [];

    // Unmount is not the only way a panel goes away. pagehide covers the tab
    // closing, which React never gets to hear about.
    const onPageHide = () => closeSession(id, identityRef.current);
    window.addEventListener("pagehide", onPageHide);

    async function open() {
      const response = await fetch(`/api/live-calls/${id}/session`, { method: "POST" });
      const body = await response.json();
      if (!response.ok) throw new Error(CONFLICTS[body.reason] ?? "Could not open this call.");
      if (cancelled) return;

      const connection = await connectMonitor({
        url: body.url,
        token: body.token,
        agentIdentity: body.agentIdentity,
      });
      if (cancelled) {
        await connection.disconnect();
        return;
      }

      connectionRef.current = connection;
      identityRef.current = body.monitorIdentity;
      setControl(connection.control);
      setLines(fromHistory(connection.history));

      cleanups.push(
        connection.on("history", (messages) =>
          setLines((current) => mergeHistory(current, messages)),
        ),
      );
      cleanups.push(
        connection.on("transcript", (segment) =>
          setLines((current) => applySegment(current, segment)),
        ),
      );
      cleanups.push(connection.on("control", setControl));
      cleanups.push(
        connection.on("disconnected", (reason) => {
          setMicOn(false);
          setError(reason ?? "The call ended.");
        }),
      );
      if (audioRef.current) cleanups.push(connection.attachAudio(audioRef.current));
    }

    open().catch((cause: unknown) => {
      setError(cause instanceof Error ? cause.message : "Could not open this call.");
    });

    return () => {
      cancelled = true;
      window.removeEventListener("pagehide", onPageHide);
      for (const cleanup of cleanups) cleanup();
      void connectionRef.current?.disconnect();
      connectionRef.current = null;
      closeSession(id, identityRef.current);
    };
  }, [id]);

  async function send(
    action: "takeover" | "end",
    extra: Record<string, string> = {},
    attempt = 0,
  ): Promise<boolean> {
    const monitorIdentity = identityRef.current;
    if (!monitorIdentity) return false;

    const response = await fetch(`/api/live-calls/${id}/${action}`, {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ monitorIdentity, ...extra }),
    });

    if (response.ok) return true;
    const body = await response.json().catch(() => ({}));

    // agent_absent is transient: the assistant is reconnecting and the call is
    // still live. not_live is final and must never be retried.
    if (body.reason === "agent_absent" && attempt < 2) {
      await new Promise((resolve) => setTimeout(resolve, 750));
      return send(action, extra, attempt + 1);
    }

    setError(CONFLICTS[body.reason] ?? `Could not ${action} this call.`);
    return false;
  }

  async function takeOver() {
    // Order matters. The token denies publishing until takeover has returned.
    if (!(await send("takeover"))) return;
    try {
      // Once only. The SDK already retries the permission race internally.
      await connectionRef.current?.setMicrophoneEnabled(true);
      setMicOn(true);
    } catch (cause) {
      if (cause instanceof MonitorError && cause.code === "mic_denied") {
        setError("Your browser blocked the microphone. Allow it, then take over again.");
      } else {
        // mic_requires_takeover reaching here means the takeover never landed.
        // That is a bug in this component, not something the person can fix.
        console.error("Opened the microphone before takeover completed", cause);
        setError("Could not open your microphone.");
      }
    }
  }

  // There is no "hand back" call to make. Releasing the microphone and
  // disconnecting is what returns the call: the assistant resumes on its own a
  // few seconds after the supervisor leaves the room. Watch the `control`
  // event for it rather than assuming the moment you disconnect.
  async function stopSpeaking() {
    await connectionRef.current?.setMicrophoneEnabled(false);
    setMicOn(false);
  }

  const holding = control === "human" || control === "bridging";

  return (
    <div>
      <p aria-live="polite">
        {control === "bridging"
          ? "Handing the call to you"
          : holding
            ? "You are speaking to the caller"
            : `Assistant is handling the call (${control})`}
      </p>

      <ol>
        {lines.map((line) => (
          <li key={line.key} data-role={line.role} data-interim={!line.final}>
            <b>{line.role}</b> {line.text}
          </li>
        ))}
      </ol>

      {!holding && <button onClick={takeOver}>Take over</button>}
      {holding && <button onClick={() => void stopSpeaking()}>Stop speaking</button>}
      {holding && <button onClick={() => void send("end", { reason: "resolved" })}>End call</button>}
      {micOn && <span>Microphone open</span>}

      <p aria-live="polite">{error}</p>
      <audio ref={audioRef} autoPlay />
    </div>
  );
}

Render it as <MonitorPanel id={row.id} />. That is row.id, never row.callId.

The media element carries the room, so it plays the caller, the assistant, and a colleague who has taken the call over. Browsers block audio that starts without a user gesture. Opening this panel from a click on a call row usually supplies the gesture; if the media element still refuses to play, offer a button that calls element.play() and try again from there. Put the caller's audio behind a mute control of your own if your staff need one, by toggling muted on the same element.

Next steps#