AI Gateway Audio Input
Send a recording with a question to a model that can understand audio. Ask for a summary or information from the recording. For a dedicated transcript, use speech to text. For a live voice conversation, use realtime.
Set AI_GATEWAY_API_KEY, install your SDK or API client, and save a short MP3 recording as greeting.mp3 in your working directory. These examples use google/gemini-3.6-flash and request text output.
Audio support depends on the selected model, provider, and request format. Confirm the accepted encoding, duration, and request-size limits for your route before using longer recordings.
These examples request a summary of the speech in the MP3 file. The SDK examples print the summary; cURL prints the JSON response containing it:
import { readFileSync } from 'node:fs';
import { generateText } from 'ai';
const { text } = await generateText({
model: 'google/gemini-3.6-flash',
messages: [
{
role: 'user',
content: [
{
type: 'text',
text: 'Summarize what the speaker says in one sentence.',
},
{
type: 'file',
data: readFileSync('greeting.mp3'),
mediaType: 'audio/mpeg',
},
],
},
],
});
console.log(text);import asyncio
from pathlib import Path
import ai
async def main():
model = ai.get_model("google/gemini-3.6-flash")
messages = [
ai.user_message(
"Summarize what the speaker says in one sentence.",
ai.file_part(Path("greeting.mp3").read_bytes(), media_type="audio/mpeg"),
)
]
async with ai.stream(model, messages) as stream:
async for event in stream:
if isinstance(event, ai.events.TextDelta):
print(event.chunk, end="", flush=True)
print()
asyncio.run(main())import { readFileSync } from 'node:fs';
import OpenAI from 'openai';
const client = new OpenAI({
apiKey: process.env.AI_GATEWAY_API_KEY,
baseURL: 'https://ai-gateway.vercel.sh/v1',
});
const data = readFileSync('greeting.mp3').toString('base64');
const response = await client.chat.completions.create({
model: 'google/gemini-3.6-flash',
messages: [
{
role: 'user',
content: [
{
type: 'text',
text: 'Summarize what the speaker says in one sentence.',
},
{
type: 'file',
file: {
filename: 'greeting.mp3',
file_data: `data:audio/mpeg;base64,${data}`,
},
},
],
},
],
});
console.log(response.choices[0]?.message.content);import base64
import os
from pathlib import Path
from openai import OpenAI
client = OpenAI(
api_key=os.environ["AI_GATEWAY_API_KEY"],
base_url="https://ai-gateway.vercel.sh/v1",
)
data = base64.b64encode(Path("greeting.mp3").read_bytes()).decode("ascii")
response = client.chat.completions.create(
model="google/gemini-3.6-flash",
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": "Summarize what the speaker says in one sentence.",
},
{
"type": "file",
"file": {
"filename": "greeting.mp3",
"file_data": f"data:audio/mpeg;base64,{data}",
},
},
],
}
],
)
print(response.choices[0].message.content)FILE_BASE64=$(base64 < greeting.mp3 | tr -d '\n')
printf '{
"model": "google/gemini-3.6-flash",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Summarize what the speaker says in one sentence."
},
{
"type": "file",
"file": {
"filename": "greeting.mp3",
"file_data": "data:audio/mpeg;base64,%s"
}
}
]
}
]
}' "$FILE_BASE64" | \
curl --fail-with-body https://ai-gateway.vercel.sh/v1/chat/completions \
-H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
-H "Content-Type: application/json" \
--data-binary @-import { readFileSync } from 'node:fs';
import OpenAI from 'openai';
const client = new OpenAI({
apiKey: process.env.AI_GATEWAY_API_KEY,
baseURL: 'https://ai-gateway.vercel.sh/v1',
});
const data = readFileSync('greeting.mp3').toString('base64');
const response = await client.responses.create({
model: 'google/gemini-3.6-flash',
input: [
{
role: 'user',
content: [
{
type: 'input_text',
text: 'Summarize what the speaker says in one sentence.',
},
{
type: 'input_file',
filename: 'greeting.mp3',
file_data: `data:audio/mpeg;base64,${data}`,
},
],
},
],
});
console.log(response.output_text);import base64
import os
from pathlib import Path
from openai import OpenAI
client = OpenAI(
api_key=os.environ["AI_GATEWAY_API_KEY"],
base_url="https://ai-gateway.vercel.sh/v1",
)
data = base64.b64encode(Path("greeting.mp3").read_bytes()).decode("ascii")
response = client.responses.create(
model="google/gemini-3.6-flash",
input=[
{
"role": "user",
"content": [
{
"type": "input_text",
"text": "Summarize what the speaker says in one sentence.",
},
{
"type": "input_file",
"filename": "greeting.mp3",
"file_data": f"data:audio/mpeg;base64,{data}",
},
],
}
],
)
print(response.output_text)FILE_BASE64=$(base64 < greeting.mp3 | tr -d '\n')
printf '{
"model": "google/gemini-3.6-flash",
"input": [
{
"role": "user",
"content": [
{
"type": "input_text",
"text": "Summarize what the speaker says in one sentence."
},
{
"type": "input_file",
"filename": "greeting.mp3",
"file_data": "data:audio/mpeg;base64,%s"
}
]
}
]
}' "$FILE_BASE64" | \
curl --fail-with-body https://ai-gateway.vercel.sh/v1/responses \
-H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
-H "Content-Type: application/json" \
--data-binary @-See the AI SDK file-part guide and Python beta multimodal-message guide for constructing SDK messages. The table below compares those messages with the HTTP formats.
| Format | Input representation | Details |
|---|---|---|
| AI SDK 7 | file part with mediaType: 'audio/mpeg' | Pass bytes, a data URL, or a supported URL. See file parts. |
| AI SDK for Python (beta) | ai.file_part(..., media_type="audio/mpeg") | See Python file parts. |
| Chat Completions | file with filename and a file_data data URL | AI Gateway maps the file to the Gemini provider. The Chat Completions endpoint rejects input_audio parts. Use file with a model that accepts audio. |
| Messages API | No audio content part | Use one of the other formats shown here, or transcribe first and send the transcript as text. |
| Responses / OpenResponses | input_file with filename and a file_data data URL | For non-OpenAI models, AI Gateway translates the request through OpenResponses. Native Responses models have their own input support. |
These examples use AI Gateway's compatible endpoints with a Gemini model. They do not imply that the same file works with every OpenAI model or the direct OpenAI API. See API format differences before changing models or endpoints.
Inline base64 increases the request size. Check the selected model's context and file-size limits, and keep any fallback models compatible with the input type. When using a hosted URL, make sure the provider can retrieve it without your application's browser session.
For request details and failures, inspect AI Gateway logs. See Inputs & Tools to choose another capability.
Was this helpful?