When an embedding model changes, existing vectors usually need to be regenerated. You cannot safely compare query embeddings from the new model against stored document embeddings from the old model as if they live in the same vector space. Even if both vectors have the same dimension, their coordinates may represent different semantic relationships.
The safest production approach is to treat an embedding model change as an index migration. Keep the old search path live, build the new embeddings in parallel, evaluate the new index, switch traffic only when it passes, and keep a rollback path until the migration is proven.
Why Re-Embedding Is Necessary
An embedding model defines the vector space used by your search system. Stored document vectors and query vectors must be produced by compatible models and preprocessing. If you change the model for new queries but leave old document vectors in place, similarity search can become unreliable.
This can break semantic search, recommendations, clustering, and RAG retrieval. The system may return documents that look numerically close but are not meaningfully relevant. For high-stakes applications, this can create silent answer-quality regressions.
The Zero-Downtime Goal
Zero downtime does not mean the migration has no cost. It means users can continue searching while the new embeddings are generated, tested, and promoted.
The migration should meet these goals:
- The old index keeps serving production traffic during backfill.
- The new index is built without corrupting or partially replacing the old one.
- Queries can be tested against both old and new embeddings before cutover.
- Cutover is fast and reversible.
- Rollback does not require another full re-embedding job.
- Documents, chunks, model versions, and search settings are auditable.
Recommended Pattern: Build a Parallel Index
The cleanest pattern is to build a parallel index or collection for the new embedding model. The old index remains the production target. The new index receives the same source content, metadata, permissions, and chunk identifiers, but stores embeddings from the new model.
This avoids mixing incompatible vectors. It also gives you a simple rollback path: keep the old index available until the new one has passed evaluation and production monitoring.
In databases that support collection aliases, the application can query a stable alias such as KnowledgeBaseProduction. During migration, that alias points to the old collection. After validation, the alias is updated to point to the new collection. If problems appear, the alias can be pointed back to the old collection.
Migration Flow
A safe re-embedding migration usually follows this sequence:
- Record the current production embedding model, chunking logic, index settings, and retrieval settings.
- Create a new index, collection, or vector space for the new model.
- Backfill existing content into the new target and generate new embeddings.
- Keep source IDs, chunk IDs, permissions, metadata, and timestamps aligned with production.
- Run offline evaluation against a labeled query set.
- Run shadow queries or side-by-side comparisons with production traffic.
- Fix chunking, metadata, ranking, or model issues before cutover.
- Switch traffic using an alias, routing flag, or deployment configuration.
- Monitor relevance, latency, errors, and user feedback.
- Keep the old index for rollback until the new index is stable.
Do Not Change Everything at Once
An embedding model migration is already a major retrieval change. If you also change chunking, metadata, filters, hybrid weighting, reranking, and prompt format at the same time, it becomes difficult to know which change caused a regression.
When possible, keep one controlled migration axis. First compare the old and new embedding model using the same source content and similar chunking. If you must change chunking too, label the migration as a broader index-generation change and evaluate it accordingly.
Track Embedding Version Metadata
Every retrievable object should carry version metadata. This makes migrations auditable and helps prevent mixed-index mistakes.
Useful fields include:
embedding_model: model name or provider identifier.embedding_model_version: exact model version or release date.embedding_dimension: vector dimension.chunking_strategy: the chunking method and version.source_document_id: stable source ID.chunk_id: stable chunk ID or deterministic hash.indexed_at: time the vector was generated.index_generation: migration generation such askb-2026-06-v2.
This metadata is especially important for RAG systems, where a retrieved chunk may be used as evidence in a generated answer.
Backfill Without Blocking Production
Backfill should run in batches while the current production index remains live. The batch job should be restartable, observable, and idempotent. If it fails halfway through, it should resume from known source IDs rather than starting over blindly.
Track progress by source document count, chunk count, failed objects, embedding API errors, indexing latency, and queue depth. Large migrations should also rate-limit embedding calls and database writes so the backfill does not degrade production traffic.
Keep Metadata and Permissions in Sync
The new index must preserve the same access boundaries as the old one. A model upgrade should not accidentally expose private documents or cross-tenant data.
During backfill, copy permissions, tenant IDs, document status, language, product, region, source URL, and freshness metadata. If permissions can change while the migration is running, design a sync step that applies updates to both old and new targets until cutover is complete.
Evaluate Before Cutover
Do not switch because the new model has better public benchmark scores. Evaluate it on your own content and query patterns.
Use a test set with real queries and expected results. Include exact lookups, broad semantic questions, short phrases, long natural-language questions, RAG questions, filtered queries, and edge cases with acronyms or identifiers.
Compare old and new indexes using metrics such as precision@k, recall@k, MRR, nDCG@k, latency, and retrieval coverage. For RAG, check whether the retrieved context contains enough evidence to answer safely.
Shadow Test With Live Query Patterns
After offline evaluation, run shadow tests. Production users still receive results from the old index, but the same queries are also sent to the new index in the background. Store both result sets for analysis.
Shadow testing helps reveal differences that offline query sets miss. Look for result overlap, ranking changes, missing exact matches, latency spikes, filter issues, and cases where the new model retrieves semantically related but unsupported content.
Cutover and Rollback
The cutover should be small and reversible. Avoid a deployment that rewrites application code in many places. Prefer a routing layer, feature flag, index alias, or collection alias.
For example, Weaviate collection aliases can point application queries at a stable alias while the underlying collection changes. The application keeps using the alias, and the alias can be updated to point from the old collection to the new one. If something goes wrong, the alias can be switched back.
Rollback should be tested before production cutover. A rollback plan that has never been tested is only a hope.
Alternative Pattern: Add a New Named Vector
Some databases support multiple vectors on the same object. This can be useful for experimentation because old and new embeddings can coexist on one record. Queries can target the old vector or the new vector explicitly.
This approach can help with side-by-side testing, but it may not be the cleanest production migration path. It can increase storage costs, complicate query code, and make cleanup harder if the old vector cannot be removed later. Use it when the database model and operational constraints make it appropriate, not as the default for every migration.
RAG-Specific Considerations
For RAG, embedding migration affects more than search ranking. It affects which evidence reaches the language model. A new model may improve semantic recall while also retrieving broader, less directly quotable context. That can change answer faithfulness.
Evaluate RAG behavior separately from search relevance. For each test question, check whether the retrieved chunks contain the required evidence, whether citations still point to useful sources, and whether the generated answer remains grounded.
If the new model changes chunk-level retrieval patterns, you may need to adjust top-k, reranking, relevance thresholds, or context selection logic.
Common Mistakes
The first mistake is mixing old document vectors with new query vectors. That creates an incompatible vector-space problem.
The second mistake is deleting the old index too early. Keep it until production monitoring shows the new index is stable.
The third mistake is skipping evaluation because the new model is newer. Better public benchmarks do not guarantee better retrieval on your corpus.
The fourth mistake is ignoring exact search behavior. A new semantic model may improve broad queries but hurt identifier-heavy or domain-specific retrieval unless hybrid search and keyword weighting are evaluated too.
Practical Checklist
- Freeze and document the current production retrieval configuration.
- Create a new index, collection, or vector namespace for the new model.
- Backfill in batches while production traffic stays on the old index.
- Store embedding model and index generation metadata.
- Run offline relevance evaluation before shadow testing.
- Run shadow queries against live traffic patterns.
- Switch traffic through an alias, feature flag, or routing layer.
- Monitor quality, latency, errors, and RAG answer grounding.
- Keep rollback available until the migration is stable.
- Archive or delete the old index only after the rollback window closes.
Practical Summary
To re-embed content when an embedding model changes without downtime, do not overwrite production vectors in place. Build a parallel index with the new model, backfill safely, evaluate retrieval quality, shadow test, cut over with a reversible routing mechanism, and keep rollback ready.
The core principle is simple: old and new embedding spaces are separate production artifacts. Treat the migration like a controlled index release, not a background cleanup job.