Most model decisions are reversible with a string change. Embedding models are not. Vectors from two different models occupy incompatible spaces, so switching means re-embedding an entire corpus, and dimension count is baked into every vector already stored. That makes the choice load-bearing on data rather than on code, which is why it deserves more than a leaderboard glance. This guide covers the six criteria that decide the fit and how to test one before committing.

Embedding models also cover images and audio, and multimodal retrieval brings its own evaluation problems. This guide is about text retrieval, where most production retrieval-augmented generation (RAG) and semantic search systems live.

## [Copy link to heading](#key-takeaways)Key takeaways

- Task type sets the decision before any other criterion, because retrieval and classification reward different models.

- Domain fit beats leaderboard rank once real query traffic depends on specialized vocabulary.

- Dimension count is schema design, since changing it later means re-embedding and re-indexing the entire corpus.

- An embedding swap costs a string change at the application layer and a full re-index at the corpus layer.

- Benchmarks combining open and private datasets expose the generalization gap that averaged leaderboard scores hide.

## [Copy link to heading](#what-are-embedding-models)What are embedding models?

An embedding model is a neural network, typically a transformer encoder, that converts text into a dense numerical vector. Each vector is a list of float values that places its input as a point in high-dimensional space, where location carries semantic meaning and the distance between two points measures how related their inputs are.

A pooling layer compresses token-level vectors into one fixed-size vector for indexing, and that fixed size is the dimension count. The model that produces those vectors defines the geometry retrieval, which depends on the geometry being fixed the moment the corpus is embedded.

### [Copy link to heading](#how-embedding-models-differ-from-the-chat-models-teams-already-run)How embedding models differ from the chat models teams already run

Most teams meet embedding models after they have already shipped something with a chat model, and they carry over the wrong mental model of what a swap costs. Changing a chat model changes what happens on the next request. Changing an embedding model invalidates everything already stored.

The comparison worth studying is where each decision leaves residue:

| Dimension | Chat model | Embedding model |
| --- | --- | --- |
| What the call returns | Tokens consumed by the user or the next step | A fixed-length vector written to an index |
| What persists after the call | Nothing beyond logs and cached responses | Every vector, for the life of the corpus |
| Cost of switching | A string change, effective on the next request | A string change plus re-embedding and re-indexing the full corpus |
| What a version bump breaks | Output style, formatting, and prompt sensitivity | Vector comparability, since new vectors do not align with old ones |
| How a poor choice surfaces | Visible in responses during development | Silently, as retrieval that returns plausible but wrong sources |
| What a benchmark predicts | General capability across many task types | Only the task type measured, on the corpora measured |

The last two rows drive the rest of this guide. A weak embedding model fails quietly, and a general benchmark score is a weak predictor of how it will fail on a specific corpus.

### [Copy link to heading](#why-embedding-model-choice-matters-for-production-teams)Why embedding model choice matters for production teams

Model choice sets five properties of the retrieval system that are hard to change once traffic is running, and each traces back to that fixed geometry:

- **Domain geometry:** A general-purpose model interprets specialized vocabulary through a broad training distribution, which changes how internal enterprise terms or biomedical terms cluster relative to each other.

- **Vector space incompatibility:** Queries and documents have to pass through the same model, because two models place the same text in unrelated coordinate systems.

- **Exact-match behavior:** Dense embeddings handle identifiers such as `SKU-7821-B` poorly, so hybrid retrieval needs to stay available to keep exact operational phrases reachable.

- **Schema lock-in:** Dimension count is written into every stored vector, so changing it means regenerating and reloading the whole corpus.

- **Grounding quality:** Retrieval that reliably surfaces the right source gives the generation step a stronger grounding layer, so an unmeasured default hides a production tradeoff rather than avoiding one.

None of these properties announce themselves during a prototype with 500 documents. Measured retrieval on a real corpus turns all five into engineering decisions with visible costs, which is what explicit criteria are for.

## [Copy link to heading](#six-criteria-for-embedding-model-selection)Six criteria for embedding model selection

Model lists date quickly while selection criteria don't. What follows applies to today's models and to ones released after this article.

Before any of the six, settle what the model is for. Every embedding leaderboard sorts by task, because a model tuned for classification and one tuned for retrieval are different decisions with different winners. RAG and semantic search are retrieval problems, so the retrieval score is the one that carries information and the headline average is the one that hides it. Teams that skip this step compare models on a number that partly measures clustering and summarization performance they will never use.

### [Copy link to heading](#1.-dimension-count)1\. Dimension count

Dimension count behaves like schema. It belongs in the design conversation before the first ingest run, not in a round of tuning afterward.

Each `float32` dimension costs 4 bytes, and storage scales linearly from there. Ten million vectors need roughly 14.3 GiB at 384 dimensions, 28.6 GiB at 768, 57.2 GiB at 1,536, and 114.4 GiB at 3,072. Index memory tends to grow alongside the raw vectors, so the practical gap between a compact model and a large one widens as the corpus does. Smaller vectors also make each similarity computation cheaper, though how much recall that costs is a question only the target corpus answers.

Three levers control storage cost, and they combine:

- **Dimension count at selection time:** The base decision, and the one that hardens into schema.

- **Matryoshka representation learning (MRL) truncation:** Models trained with nested representations tolerate having their vectors shortened after generation, degrading gracefully rather than collapsing as dimensions are dropped.

- **Quantization:** Reducing the precision of each value rather than the number of values. Moving from `float32` to `int8` cuts storage 4x, and binary quantization to one bit per dimension cuts it 32x, with a rescoring pass over the top candidates recovering most of the lost recall.

Quantization is a post-processing step on the vectors, so a team can revisit it later in a way they cannot revisit the dimension count. Any change to a quantization scheme still needs a fresh eval on the target corpus, because the recall cost is corpus-dependent. A mid-sized vector from an MRL-capable model is a defensible starting point, since it preserves both remaining levers.

### [Copy link to heading](#2.-benchmark-rank)2\. Benchmark rank

The [Massive Text Embedding Benchmark (MTEB)](https://arxiv.org/pdf/2210.07316) works as a shortlist filter and fails as a decision procedure. It shows which models are competitive in general, and the corpus decides which one wins.

Two structural problems limit what a headline rank can convey. The current multilingual edition, MMTEB, spans over 500 evaluation tasks across 250-plus languages and 10 task categories, and averaging across categories dilutes the retrieval signal that predicts RAG performance. The retrieval suite also leans on general web corpora such as MS MARCO, which resemble a specialized enterprise corpus only loosely.

Benchmark overfitting is the harder problem. When training data overlaps with evaluation data, scores inflate without improving generalization, and the leaderboard rewards models for memorizing the test set. The [Retrieval Embedding Benchmark (RTEB)](https://huggingface.co/blog/rteb) was built to detect exactly that, using a hybrid of open datasets anyone can reproduce and private datasets scored by the maintainers.

A model that scores well on the open half and drops on the private half directly reveals its generalization gap. For a team whose corpus the model has never seen, that gap carries more information than the rank above it. RTEB covers law, healthcare, code, and finance across 20 languages, and reports normalized discounted cumulative gain at rank 10 (`nDCG@10`) rather than a blended average.

RTEB is in beta and names its own limitations, and one of them deserves attention. Around half of its retrieval datasets are repurposed from question-answering sets, which can create lexical overlap between query and document and quietly favor keyword matching over semantic understanding. Use it as a stronger filter than a general average, not as a verdict. The multimodal equivalent for teams whose retrieval spans images and video is the Massive Multimodal Embedding Benchmark (MMEB).

Both benchmarks narrow the field. Neither replaces an eval set built from a team's own documents, and a top-ranked general model losing to a lower-ranked one on an in-domain set is a routine result rather than a surprising one.

### [Copy link to heading](#3.-domain-and-language-fit)3\. Domain and language fit

Specialized models win when the corpus uses specialized language, and the effect is large enough to show up in the foundational out-of-domain research.

The [BEIR benchmark](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/file/65b9eea6e1cc6bb9f0cd2a47751a186f-Paper-round2.pdf) evaluated 10 retrieval systems across 18 datasets and found that BM25, a bag-of-words lexical baseline, remains a strong zero-shot performer, while dense and sparse retrieval models trained on MS MARCO can substantially underperform it on data they were not trained for. Vendors have responded to that gap commercially, and domain-tuned embedding models now ship as products for legal, financial, and code retrieval alongside general-purpose ones. In production the mismatch is rarely subtle. It looks like a model that fails to resolve MI as myocardial infarction, and it surfaces as a retrieval miss rather than an error.

Language fit follows the same logic, and English results do not transfer. Multilingual traffic needs the languages it actually serves tested directly, using queries in those languages against documents in those languages.

A domain-specific model for the corpus belongs on the shortlist by default. If its in-domain gain sits inside eval noise, the general model wins on lower migration risk and broader provider support, which is a real advantage rather than a consolation.

### [Copy link to heading](#4.-cost-and-latency-at-volume)4\. Cost and latency at volume

Per-token pricing looks negligible until it runs across a few million documents. The first full ingest is expensive at any rate, and every re-index pays it again.

At query time, latency is the binding constraint. Real-time search runs on a tight end-to-end budget, and the embedding call has to fit inside it, so a cross-region call that eats most of that budget before retrieval even starts is a real problem. Caching frequent query vectors and dropping to a smaller model are the two levers, and a retrieval metric only means something read next to a latency measurement taken from the same region.

### [Copy link to heading](#5.-open-weights-versus-hosted-api)5\. Open weights versus hosted API

Model lifecycle risk runs higher for embeddings than for chat, because replacement vectors don't line up with the vectors already in the index. When a provider retires an embedding model, the corpus needs re-embedding before the replacement can serve the same index, and the deprecation clock is set by someone else.

Open-weight models such as Nomic Embed reduce that exposure. A team can pin a checkpoint indefinitely, and the text never leaves its own infrastructure, which matters when regulated workloads require that control.

Privacy is a sharper consideration than data residency alone. [Research on embedding inversion](https://arxiv.org/abs/2310.06816) has shown that vectors are not a safe one-way transformation of their inputs. A multi-step method that iteratively corrects and re-embeds candidate text recovered 92% of 32-token inputs exactly, and recovered full names from a dataset of clinical notes. Treat a vector index holding sensitive text with the access controls a team would apply to the text itself, whoever hosts the model.

At low volume, hosted APIs win on cost and operational overhead, and self-hosting an encoder to save on a small ingest is work without a return. At high sustained volume with utilization to match and MLOps capacity already in place, self-hosting becomes economically rational. The decision turns on sustained utilization rather than peak throughput.

### [Copy link to heading](#6.-input-length-limits)6\. Input length limits

Input caps vary by more than an order of magnitude across current models, and that spread sets the chunking strategy rather than the reverse.

Compact sentence-transformer and BGE-style models commonly cap input at 512 tokens. Newer hosted models accept far more, with current gateway-addressable models ranging from 32,000 tokens up to 128,000. Over-limit behavior is provider-specific, and the difference matters. Some providers truncate silently and return a vector built from partial content, while others reject the request outright. Silent truncation is the more dangerous of the two, because the pipeline reports success and the dropped tokens never reach the index.

General benchmarks give no warning here, since much of the retrieval suite uses short passages and a model that degrades on long inputs still scores well. Measuring the token-length distribution of real documents comes first, because chunk size cannot exceed the model's maximum. Each of these six criteria is only as good as the eval set behind it.

## [Copy link to heading](#four-practices-for-evaluating-embedding-models-before-committing)Four practices for evaluating embedding models before committing

All four run before the first production index exists, while changing course still costs a rerun rather than a migration. Running them early is what lets a team weigh quality against cost while the schema is still soft.

### [Copy link to heading](#1.-build-a-domain-specific-eval-set-first)1\. Build a domain-specific eval set first

Benchmark scores rarely transfer cleanly to a specific corpus, and retrieval quality is where the gap shows. A working eval set pairs real queries with the documents that should answer them, and even a small, carefully curated set beats none.

Pick metrics that match what the team needs to know:

- `**Recall@k**`**:** Whether relevant documents appear at all in the first k results, for binary relevance.

- `**NDCG**`**:** Graded relevance, which rewards relevant documents more when they rank higher.

- `**MRR**` **(mean reciprocal rank):** How high the first correct result lands, which is what matters when users read one answer.

- `**MAP**` **(mean average precision):** Precision across the full ranked list, for cases where several documents are relevant.

Add a robustness pass over edge cases and deliberately diverse inputs, including short queries, long queries, identifiers, and mixed-language text. Aggregate scores hide the inputs where a model collapses, and those inputs tend to be the ones users send.

### [Copy link to heading](#2.-test-with-real-production-queries,-not-sample-text)2\. Test with real production queries, not sample text

An eval set built by generating questions from the chunks they retrieve is misleading, because those questions sit unnaturally close to their source and make retrieval look easier than it will be in production. Queries pulled from real logs, search history, and support tickets are harder, and they are the honest test.

Set the threshold before testing rather than after seeing results. If a candidate misses an `nDCG@10` or `Recall@k` target, a cross-encoder reranker is the cheaper next move, since it improves ranking without touching stored vectors. Fine-tuning comes after that, and it adds a model the team then owns.

### [Copy link to heading](#3.-price-the-re-embedding-before-locking-a-dimension-count)3\. Price the re-embedding before locking a dimension count

Dimension count locks in on the first ingest, and changing it means regenerating and reloading every vector. The token bill is usually the smallest line in that estimate. Engineering time, index rebuild windows, and the risk carried by a cutover cost more.

An MRL-capable model keeps truncation available, which is the option that matters when the first index succeeds and the corpus grows faster than the storage plan assumed. Write the estimate down before committing, because a number produced under deadline pressure later will be optimistic.

### [Copy link to heading](#4.-plan-the-model-migration-before-it-becomes-urgent)4\. Plan the model migration before it becomes urgent

Vector database vendors make a reasonable case that model choice doesn't have to be permanent: Qdrant notes that its architecture makes migrating between models relatively straightforward, and Weaviate recommends starting with a lightweight model and swapping later. That advice is accurate about the layer it describes, the storage layer, which is worth reconciling with the position here rather than treating as a contradiction.

Both are true at different layers. A swap is close to free at the application layer, where the model reference is a string in a config file. It is expensive at the corpus layer, where every vector has to be regenerated and re-indexed before the new model can answer a single query. The teams that get surprised are the ones who priced the application-layer change and scheduled the corpus-layer one for a sprint that turned out to be a quarter.

Two patterns keep the corpus-layer cost manageable:

- **Versioned metadata on every embedding:** Store the model name, version, dimension count, distance metric, and any required prefix format alongside each vector. The cost is negligible and it makes the eventual swap traceable instead of archaeological.

- **Multiple vectors per record:** Vector stores that support several named vectors on one record, each with its own dimension count and distance metric, allow a second model to be added to an existing collection without recreating it. Queries can then route to one model or run hybrid retrieval across both during a coexistence period, which replaces a cutover with a gradual shift.

A blue-green pattern covers the case where coexistence isn't practical, validating the new model and index against the same eval set before any traffic moves, with the old index live until the new one proves out. Either way, the migration is a planned project with a schedule rather than an emergency.

## [Copy link to heading](#how-vercel-helps-engineering-teams-put-an-embedding-model-into-production)How Vercel helps engineering teams put an embedding model into production

Selection criteria and migration plans both assume reversibility, and reversibility is largely a property of the infrastructure the calls run through. The [AI Gateway](https://vercel.com/docs/ai-gateway) and surrounding primitives are built so provider choice doesn't harden into application code.

### [Copy link to heading](#swap-embedding-models-with-a-config-change-through-ai-gateway)Swap embedding models with a config change through AI Gateway

Committing to one provider's SDK makes the swap larger than the model decision that triggered it, and the rewrite lands on whichever team owns the ingest pipeline. AI Gateway exposes one API across providers, so moving between models and providers doesn't require rewriting application code.

Embedding models are addressed as `creator/model-name` strings, which puts dozens of embedding models from different providers behind one interface and one API key. Moving to the shortlist winner becomes a string change at the application layer, leaving the eval set and the migration plan to handle the corpus-layer work that actually carries risk.

### [Copy link to heading](#batch-and-single-embedding-calls-with-the-ai-sdk)Batch and single embedding calls with the AI SDK

Hand-rolled provider calls make ingest pipeline behavior harder to replace later, because retry logic, batching, and error handling end up specific to one vendor's client. [AI SDK](/blog/ai-sdk-7) gives TypeScript teams one interface for both call shapes, with `embed` for single values and [`embedMany`](https://vercel.com/docs/ai-gateway/modalities/embeddings) for batches.

A batch ingest call looks like this:

```
import { embedMany } from 'ai';

const { embeddings } = await embedMany({
  model: 'openai/text-embedding-3-small',
  values: chunks,
});
```

Keeping batch size and concurrency visible next to token usage and the model string is what makes ingest throughput and query latency measurable while the eval set decides which model serves production traffic.

### [Copy link to heading](#keep-ingest-pipelines-running-through-provider-outages)Keep ingest pipelines running through provider outages

A document is not searchable until its embedding call finishes, so a provider outage midway through a large ingest leaves the corpus half-indexed and a queue to reconcile by hand. AI Gateway routes across providers through one endpoint and retries automatically when a provider fails.

Teams can also configure [model fallbacks](https://vercel.com/docs/ai-gateway/models-and-providers/model-fallbacks) that are tried in sequence until one succeeds. Ordering is worth thinking about for embeddings specifically, since a fallback to a different model produces vectors that don't match the index. Falling back across providers hosting the same model keeps vectors comparable, while falling back to a different model is a decision to make deliberately rather than by default.

### [Copy link to heading](#run-full-corpus-re-embedding-as-a-durable-workflow)Run full-corpus re-embedding as a durable workflow

Re-embedding a corpus does not fit inside a request, and the first attempt often reveals that by timing out halfway through. Incremental ingest and request-sized embedding work fit Fluid compute, where [function duration](https://vercel.com/docs/functions/configuring-functions/duration) defaults to 300s and reaches 800s on Pro and Enterprise, with an extended maximum of 1,800s currently in beta for per-function configuration.

Full-corpus migrations need orchestration that survives a deploy and resumes from a checkpoint instead of starting over. [Vercel Workflows](https://vercel.com/docs/workflows) does this, holding state for minutes to months with no duration limit and surviving crashes or deployments mid-migration. Keeping that orchestration outside the user request path, then cutting traffic over only after the eval set confirms retrieval quality on the new index, is what turns a re-embedding run into a routine operation.

## [Copy link to heading](#choose-and-route-embedding-models-on-vercel)Choose and route embedding models on Vercel

Choosing an embedding model looks like a model decision and is really a decision about which layer absorbs the next change. Dimension count, task type, and domain fit set retrieval quality, and all three are reversible only through a full corpus rebuild. What the platform underneath decides is whether the application-layer half of a swap costs a config edit or a rewrite. That is why model evaluation and platform choice belong in the same conversation.

Vercel gives teams the primitives that keep an embedding choice reversible:

- **AI Gateway:** Embedding providers addressed as `creator/model-name` strings behind one API key, so the shortlist winner is a config change rather than a rewrite.

- **Model fallbacks:** An ordered provider list that keeps an ingest pipeline running when one embedding provider degrades.

- **AI SDK:** One interface for `embed` and `embedMany` across providers, so ingest and query paths don't carry vendor-specific client code.

- **Vercel Workflows:** Durable execution with checkpoints and retries for full-corpus re-embedding, with no duration limit and state that survives deploys.

- **AI Gateway observability:** Token counts and latency by model, so ingest throughput stays visible while a migration runs.

To put a shortlisted embedding model into production, [start a new project](https://vercel.com/new) and keep the model path replaceable from the first commit, or browse [Vercel templates](https://vercel.com/templates) for a working starting point.

## [Copy link to heading](#frequently-asked-questions-about-embedding-models)Frequently asked questions about embedding models

### [Copy link to heading](#can-teams-switch-embedding-models-without-re-embedding-the-corpus)Can teams switch embedding models without re-embedding the corpus?

No. Embeddings from different models, and from different versions of one model, occupy geometrically incompatible spaces, so every switch requires re-embedding the corpus. A blue-green cutover that validates the new index before routing queries to it keeps the swap reversible.

### [Copy link to heading](#are-more-embedding-dimensions-always-better)Are more embedding dimensions always better?

Not automatically. The right dimension count is workload-specific, so test candidates against the target corpus rather than defaulting to the maximum. An MRL-capable model leaves room to shorten vectors later without regenerating them, and quantization reduces storage further without changing dimension count.

### [Copy link to heading](#how-much-notice-do-providers-give-before-deprecating-embedding-models)How much notice do providers give before deprecating embedding models?

Notice varies by provider and by where a model sits in its lifecycle. Because replacement vectors don't align with existing ones, plan for full re-embedding and re-indexing whenever a provider retires an embedding model, and prefer pinnable open weights when that timeline has to stay under a team's control.

### [Copy link to heading](#do-open-source-embedding-models-need-special-query-prefixes)Do open-source embedding models need special query prefixes?

Often, yes. Several open model families require task prefixes, such as distinct markers for queries and documents, and omitting a required prefix degrades retrieval quality without raising an error. Check the model card, then store the prefix format in the same versioned metadata as the model name and dimensions.