Skip to content
Docs

AI Gateway Model Fallbacks

You can configure model failover to specify backups that are tried in order if the primary model fails or is unavailable.

Using the models option

Add a models array to providerOptions.gateway to list fallback models. The same option works across every AI Gateway API format. Select your API below:

String entries on this page react to execution failure. On a decision request, a string entry can also name a language model, which answers through structured output. Decision requests can also add one conditional object that reacts to a successful but uncertain result. AI Gateway rejects a conditional object with a 400 error on any other request type, and when the conditional object names the primary model itself. See Decision Fallbacks for confidence, Boolean probability, compound conditions, and full-request reruns.

These examples use AI SDK 7 and the AI SDK for Python beta. Set AI_GATEWAY_API_KEY before running them. See API format differences for setup, request fields, and response handling.

See the AI SDK model-fallback reference for SDK configuration and usage.

model-fallbacks.ts
import { generateText } from 'ai';
 
const { text } = await generateText({
  model: 'anthropic/claude-fable-5',
  prompt: 'Write a haiku about TypeScript.',
  providerOptions: {
    gateway: {
      models: ['anthropic/claude-opus-5', 'google/gemini-3.1-pro-preview'],
    },
  },
});
 
console.log(text);
model-fallbacks_ai.py
import asyncio
import ai
 
async def main():
    model = ai.get_model("anthropic/claude-fable-5")
    messages = [ai.user_message("Write a haiku about TypeScript.")]
    params = ai.InferenceRequestParams(
        extra_body={"providerOptions": {"gateway": {"models": ["anthropic/claude-opus-5", "google/gemini-3.1-pro-preview"]}}}
    )
    async with ai.stream(model, messages, params=params) as stream:
        async for event in stream:
            if isinstance(event, ai.events.TextDelta):
                print(event.chunk, end="", flush=True)
    print()
 
asyncio.run(main())
model-fallbacks-chat.ts
import OpenAI from 'openai';
 
const client = new OpenAI({
  apiKey: process.env.AI_GATEWAY_API_KEY,
  baseURL: 'https://ai-gateway.vercel.sh/v1',
});
 
const response = await client.chat.completions.create({
  model: 'anthropic/claude-fable-5',
  messages: [
    {
      role: 'user',
      content: 'Write a haiku about TypeScript.',
    },
  ],
  // AI Gateway extension fields are not included in the upstream SDK types.
  ...{
    providerOptions: {
      gateway: {
        models: ['anthropic/claude-opus-5', 'google/gemini-3.1-pro-preview'],
      },
    },
  },
});
 
console.log(response.choices[0]?.message.content);
model-fallbacks_chat.py
import os
from openai import OpenAI
 
client = OpenAI(
    api_key=os.environ["AI_GATEWAY_API_KEY"],
    base_url="https://ai-gateway.vercel.sh/v1",
)
 
response = client.chat.completions.create(
    model="anthropic/claude-fable-5",
    messages=[{"role": "user", "content": "Write a haiku about TypeScript."}],
    extra_body={"providerOptions": {"gateway": {"models": ["anthropic/claude-opus-5", "google/gemini-3.1-pro-preview"]}}},
)
 
print(response.choices[0].message.content)
model-fallbacks-chat.sh
curl --fail-with-body https://ai-gateway.vercel.sh/v1/chat/completions \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "anthropic/claude-fable-5",
  "messages": [
    {
      "role": "user",
      "content": "Write a haiku about TypeScript."
    }
  ],
  "providerOptions": {
    "gateway": {
      "models": [
        "anthropic/claude-opus-5",
        "google/gemini-3.1-pro-preview"
      ]
    }
  }
}'
model-fallbacks-messages.ts
import Anthropic from '@anthropic-ai/sdk';
 
const client = new Anthropic({
  apiKey: process.env.AI_GATEWAY_API_KEY,
  baseURL: 'https://ai-gateway.vercel.sh',
});
 
const response = await client.messages.create({
  model: 'anthropic/claude-fable-5',
  messages: [
    {
      role: 'user',
      content: 'Write a haiku about TypeScript.',
    },
  ],
  max_tokens: 1024,
  ...{
    providerOptions: {
      gateway: {
        models: ['anthropic/claude-opus-5', 'google/gemini-3.1-pro-preview'],
      },
    },
  },
});
 
for (const block of response.content) {
  if (block.type === 'text') console.log(block.text);
}
model-fallbacks_messages.py
import os
from anthropic import Anthropic
 
client = Anthropic(
    api_key=os.environ["AI_GATEWAY_API_KEY"],
    base_url="https://ai-gateway.vercel.sh",
)
 
response = client.messages.create(
    model="anthropic/claude-fable-5",
    messages=[{"role": "user", "content": "Write a haiku about TypeScript."}],
    max_tokens=1024,
    extra_body={"providerOptions": {"gateway": {"models": ["anthropic/claude-opus-5", "google/gemini-3.1-pro-preview"]}}},
)
 
for block in response.content:
    if block.type == "text":
        print(block.text)
model-fallbacks-messages.sh
curl --fail-with-body https://ai-gateway.vercel.sh/v1/messages \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
  "model": "anthropic/claude-fable-5",
  "messages": [
    {
      "role": "user",
      "content": "Write a haiku about TypeScript."
    }
  ],
  "max_tokens": 1024,
  "providerOptions": {
    "gateway": {
      "models": [
        "anthropic/claude-opus-5",
        "google/gemini-3.1-pro-preview"
      ]
    }
  }
}'
model-fallbacks-responses.ts
import OpenAI from 'openai';
 
const client = new OpenAI({
  apiKey: process.env.AI_GATEWAY_API_KEY,
  baseURL: 'https://ai-gateway.vercel.sh/v1',
});
 
const response = await client.responses.create({
  model: 'anthropic/claude-fable-5',
  input: 'Write a haiku about TypeScript.',
  ...{
    providerOptions: {
      gateway: {
        models: ['anthropic/claude-opus-5', 'google/gemini-3.1-pro-preview'],
      },
    },
  },
});
 
console.log(response.output_text);
model-fallbacks_responses.py
import os
from openai import OpenAI
 
client = OpenAI(
    api_key=os.environ["AI_GATEWAY_API_KEY"],
    base_url="https://ai-gateway.vercel.sh/v1",
)
 
response = client.responses.create(
    model="anthropic/claude-fable-5",
    input="Write a haiku about TypeScript.",
    extra_body={"providerOptions": {"gateway": {"models": ["anthropic/claude-opus-5", "google/gemini-3.1-pro-preview"]}}},
)
 
print(response.output_text)
model-fallbacks-responses.sh
curl --fail-with-body https://ai-gateway.vercel.sh/v1/responses \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "anthropic/claude-fable-5",
  "input": "Write a haiku about TypeScript.",
  "providerOptions": {
    "gateway": {
      "models": [
        "anthropic/claude-opus-5",
        "google/gemini-3.1-pro-preview"
      ]
    }
  }
}'

Combining with provider routing

You can use models together with order to control both model failover and provider preference:

app/api/chat/route.ts
import { streamText } from 'ai';
 
export async function POST(request: Request) {
  const { prompt } = await request.json();
 
  const result = streamText({
    model: 'openai/gpt-6-astra',
    prompt,
    providerOptions: {
      gateway: {
        models: ['openai/gpt-5.4-nano', 'anthropic/claude-opus-5'],
        order: ['azure', 'openai'], // Provider preference for each model
      },
    },
  });
 
  return result.toUIMessageStreamResponse();
}

This configuration:

  1. Tries openai/gpt-6-astra via Azure, then OpenAI
  2. If both fail, tries openai/gpt-5.4-nano via Azure first, then OpenAI
  3. If those fail, it tries anthropic/claude-opus-5 via available providers

The models and order fields both live under providerOptions.gateway, so you can combine them the same way in the Chat Completions, Messages, OpenAI Responses, and OpenResponses APIs. For all available routing fields, see Provider Options.

How failover works

When processing a request with model fallbacks:

  1. The gateway routes the request to the primary model (the model parameter)
  2. For each model, provider routing rules apply (using order or only if specified)
  3. If all providers for a model fail, the gateway tries the next model in the models array
  4. The response comes from the first successful model/provider combination

Example provider metadata with model fallbacks

The Python beta can omit routing details from its normalized message metadata. To confirm which model served a request, inspect the raw AI Gateway response or the request logs.

When model fallbacks occur, the modelAttempts array in the provider metadata shows each model that was tried. Each attempt carries two identifiers: canonicalSlug is AI Gateway's normalized model name (always creator/model-name), while modelId is the provider's own internal ID for that model on that provider (provider:model). These identifiers differ. The same canonicalSlug can be tried via several providers, each reporting its own modelId. Failed models include error details in their providerAttempts, while the successful model includes its provider attempt details:

"modelAttempts": [
  {
    "modelId": "vertex:gemini-3.1-pro-preview",
    "canonicalSlug": "google/gemini-3.1-pro-preview",
    "success": false,
    "providerAttemptCount": 2,
    "providerAttempts": [
      {
        "attemptNumber": 1,
        "provider": "vertex",
        "modelId": "vertex:gemini-3.1-pro-preview",
        "success": false,
        "credentialType": "system",
        "responseTimeMs": 15679.64,
        "error": "Internal error encountered.",
        "statusCode": 500
      },
      {
        "attemptNumber": 2,
        "provider": "google",
        "modelId": "google:gemini-3.1-pro-preview",
        "success": false,
        "credentialType": "system",
        "responseTimeMs": 284.30,
        "error": "Internal error encountered.",
        "statusCode": 500
      }
    ]
  },
  {
    "modelId": "anthropic:claude-opus-5",
    "canonicalSlug": "anthropic/claude-opus-5",
    "success": true,
    "providerAttemptCount": 1,
    "providerAttempts": [
      {
        "attemptNumber": 1,
        "provider": "anthropic",
        "modelId": "anthropic:claude-opus-5",
        "success": true,
        "credentialType": "system",
        "statusCode": 200,
        "responseTimeMs": 4521.78,
        "providerResponseId": "msg_01ABCDEFGhJKLmnOpQrStUv"
      }
    ]
  }
]

Failover happens automatically. To see which model and provider served your request, check the provider metadata.

Fallbacks in virtual models

A virtual model carries its own fallback chain, which replaces the request's models list when set. A decision virtual model can store one conditional decision fallback. A vmc/<slug> can also be a string or conditional destination in a request's fallback list, and each entry resolves independently.

Last updated October 2, 2026
Previous
Decision Fallbacks
Next
Provider Timeouts

Was this helpful?

supported.