Live calls

Live Call WebSocket

The call itself: a WebSocket (wss://) opened with the session ticket. Send microphone audio, receive the persona's voice and transcripts as JSON events.

Open wss://api.spicyapi.com/v1/realtime?session=<ticket>, the url from POST /v1/realtime/sessions, within 60 seconds of creating it. No Authorization header: the ticket is the credential, so a browser can connect directly. An unknown, used or expired ticket is a 401 before the upgrade.

Events follow the OpenAI Realtime API shape. Send JSON text frames (up to 256 KB). Accepted client events: input_audio_buffer.append (audio: base64 PCM16 mono 16 kHz, about 100 ms per chunk), input_audio_buffer.commit, input_audio_buffer.clear, response.create, response.cancel, conversation.item.create (a user message with input_text parts only, screened before it is sent), and session.update with only session.turn_detection. Persona and voice are fixed by the session. Anything else returns an error event.

Server events: session.created, session.updated, input_audio_buffer.speech_started, input_audio_buffer.speech_stopped, input_audio_buffer.committed, conversation.item.input_audio_transcription.delta and .completed (what the caller said), response.created, response.audio.delta (delta: base64 PCM16 mono 24 kHz), response.audio_transcript.delta and .done (what the persona said), response.done and error. SpicyAPI adds spicy.usage ({ turn, cost_usd, total_cost_usd }) after each billed turn and spicy.session_expired when the call reaches its time limit, after which the socket closes.

Error events carry error.code: payment_required (balance or spend limit reached; the call ends), content_blocked (a transcript failed screening; the call ends), moderation_unavailable (screening unreachable; the call ends, retry shortly), upstream_unavailable (the voice service failed), unsupported_event, unsupported_item, unsupported_field, invalid_json and invalid_event (the call continues).

GET/v1/realtimeTry it

Live Call WebSocket

JavaScript

// Browser: your server calls POST /v1/realtime/sessions and returns session.url
const { url } = await fetch("/my-api/start-call", { method: "POST" }).then((r) => r.json());
const ws = new WebSocket(url);

// Microphone: 16 kHz mono PCM16, base64, about 100 ms per event
const mic = await navigator.mediaDevices.getUserMedia({ audio: true });
const ctxIn = new AudioContext({ sampleRate: 16000 });
const src = ctxIn.createMediaStreamSource(mic);
const proc = ctxIn.createScriptProcessor(2048, 1, 1);
src.connect(proc);
proc.connect(ctxIn.destination);
proc.onaudioprocess = (e) => {
  if (ws.readyState !== WebSocket.OPEN) return;
  const f32 = e.inputBuffer.getChannelData(0);
  const i16 = new Int16Array(f32.length);
  for (let i = 0; i < f32.length; i++) i16[i] = Math.max(-1, Math.min(1, f32[i])) * 0x7fff;
  const b64 = btoa(String.fromCharCode(...new Uint8Array(i16.buffer)));
  ws.send(JSON.stringify({ type: "input_audio_buffer.append", audio: b64 }));
};

// Playback: 24 kHz mono PCM16, queued back to back
const ctxOut = new AudioContext({ sampleRate: 24000 });
let at = 0;
ws.onmessage = (msg) => {
  const ev = JSON.parse(msg.data);
  if (ev.type === "response.audio.delta") {
    const bytes = Uint8Array.from(atob(ev.delta), (c) => c.charCodeAt(0));
    const i16 = new Int16Array(bytes.buffer);
    const buf = ctxOut.createBuffer(1, i16.length, 24000);
    buf.getChannelData(0).set(Float32Array.from(i16, (s) => s / 0x8000));
    const node = ctxOut.createBufferSource();
    node.buffer = buf;
    node.connect(ctxOut.destination);
    at = Math.max(at, ctxOut.currentTime);
    node.start(at);
    at += buf.duration;
  }
  if (ev.type === "spicy.usage") console.log("call so far: $" + ev.total_cost_usd);
  if (ev.type === "error") console.warn(ev.error.code, ev.error.message);
};

101

JSON

{ "type": "session.created", "session": { "object": "realtime.session", "model": "spicy-live-1", "voice": "Serena" } }
{ "type": "input_audio_buffer.speech_started" }
{ "type": "conversation.item.input_audio_transcription.completed", "transcript": "Hey, are you still at the bar?" }
{ "type": "response.audio.delta", "delta": "AAABAP7/AgD9/w..." }
{ "type": "response.audio_transcript.done", "transcript": "Still here, polishing glasses and waiting for you." }
{ "type": "response.done", "response": { "status": "completed" } }
{ "type": "spicy.usage", "turn": 1, "cost_usd": 0.00041, "total_cost_usd": 0.00041 }

Query parameters

sessionstringrequired

The ticket, client_secret.value from POST /v1/realtime/sessions.

Response

101 · websocket

Switching Protocols

The socket, carrying JSON events in both directions. The first server event is `session.created`.

typestringrequired

Every event has a type, e.g. response.audio.delta, spicy.usage, error.

deltastring

response.audio.delta: base64 PCM16 mono 24 kHz. Transcript deltas: text.

transcriptstring

.completed and .done transcript events: the full line.

cost_usdnumber

spicy.usage: what the last turn cost, already debited. total_cost_usd is the call so far.

errorobject

error events: { type, code, message }.

Was this page helpful?

Suggest edits Raise issue