Realtime Voice with AI Gateway

Build voice agents that listen and respond in real time. With the AI SDK, you stream microphone audio to a realtime model through AI Gateway and play back its spoken replies with low latency. The AI Gateway provider exposes gateway.experimental_realtime, which serves two roles: a server-side getToken helper that mints a connection, and a realtime model that acts as a codec, translating between normalized AI SDK events and the provider's wire format.

For a step-by-step setup, see the Realtime quickstart.

For openai/gpt-live-1, use the GPT-Live guide. GPT-Live uses a separate WebSocket endpoint, continuous audio, client-managed delegation, and duration-based billing. It does not use the gateway.experimental_realtime examples on this page. Changing their model slug is not enough to connect to GPT-Live.

Realtime is for live conversation. To transcribe recorded audio, see Speech to Text; to generate spoken audio from text, see Text to Speech.

These examples use openai/gpt-realtime-2, which supports realtime connections over WebSocket. Other realtime models include google/gemini-3.8-live (latency-optimized) and google/gemini-3.8-live-extended-thinking (background reasoning while streaming audio; requires exactly one of thinkingLevel or thinkingBudget in providerOptions.google.thinkingConfig, for example { thinkingLevel: 'LOW' }). Before switching models, check that the model supports the realtime WebSocket endpoint. A successful token request does not guarantee that a model accepts a WebSocket connection.

In the browser, the useRealtime hook from @ai-sdk/react manages the WebSocket connection, microphone capture, and audio playback. Add a server route that mints a short-lived token so your API key never reaches the client:

import { gateway } from '@ai-sdk/gateway';
 
export async function POST() {
  const { token, url } = await gateway.experimental_realtime.getToken({
    model: 'openai/gpt-realtime-2',
  });
 
  return Response.json({ token, url, tools: [] });
}

Then connect from a client component. The hook fetches your token route, opens the session, and exposes connection and audio controls:

'use client';
 
import { experimental_useRealtime as useRealtime } from '@ai-sdk/react';
import { gateway } from '@ai-sdk/gateway';
import { useMemo } from 'react';
 
export default function Page() {
  const model = useMemo(
    () => gateway.experimental_realtime('openai/gpt-realtime-2'),
    [],
  );
 
  const { status, connect, startAudioCapture } = useRealtime({
    model,
    api: { token: '/api/realtime/token' },
    sessionConfig: { voice: 'alloy', turnDetection: { type: 'server-vad' } },
  });
 
  // Call connect(), then startAudioCapture(stream) with a microphone MediaStream.
  // The hook also returns disconnect, stopAudioCapture, isCapturing, and messages.
}

Outside the browser, use the realtime model as a codec to drive a WebSocket yourself. getWebSocketConfig builds the connection from the token, and serializeClientEvent and parseServerEvent translate events to and from the normalized format:

import { gateway } from '@ai-sdk/gateway';
import WebSocket from 'ws';
 
const modelId = 'openai/gpt-realtime-2';
 
const { token, url } = await gateway.experimental_realtime.getToken({
  model: modelId,
});
 
const model = gateway.experimental_realtime(modelId);
const config = model.getWebSocketConfig({ token, url });
const ws = new WebSocket(config.url, config.protocols);
 
ws.on('open', async () => {
  ws.send(
    JSON.stringify(
      await model.serializeClientEvent({
        type: 'conversation-item-create',
        item: {
          type: 'text-message',
          role: 'user',
          text: 'Say hello in one sentence.',
        },
      }),
    ),
  );
  ws.send(
    JSON.stringify(
      await model.serializeClientEvent({ type: 'response-create' }),
    ),
  );
});
 
ws.on('message', (data) => {
  const parsed = model.parseServerEvent(JSON.parse(data.toString()));
 
  for (const event of Array.isArray(parsed) ? parsed : [parsed]) {
    if (event.type === 'audio-transcript-delta') {
      process.stdout.write(event.delta);
    }
    // event.type 'audio-delta' carries base64 PCM16 audio chunks
  }
});

The Realtime quickstart shows a complete version of this script that also collects the audio and saves it as a playable file.

Pass a sessionConfig to set the voice, turn detection, and other session options. The AI SDK normalizes these and the Gateway maps them to the provider:

• voice: the voice the model speaks with, such as alloy.
• turnDetection: how the model decides you've finished speaking, such as { type: 'server-vad' } for server-side voice activity detection.
• instructions: a system prompt for the session.
• tools: realtime tool definitions the model can call.

AI Gateway enforces these limits on the realtime sessions described on this page. For GPT-Live, see its session lifecycle:

Teams also have a limit on concurrent realtime sessions. Additional connection attempts beyond the limit are rejected until a session ends.

• Image input is not supported in realtime sessions.
• Reconnecting does not resume a previous session. Start a new session and replay any context you need.