Approximate nearest neighbor indexes are designed to return good nearest-neighbor results quickly, not to behave exactly like a brute-force scan. Because of that, ANN search results can show behaviors that look surprising if you expect strict database-style ordering.
These behaviors are usually a result of the recall, latency, filtering, scoring, and pagination trade-offs built into large-scale vector search.
Short Answer
Common ANN index search result behaviors include:
- results may not exactly match brute-force top-k
- larger limits may find better candidates
- small score differences can produce unstable ordering
- equal-distance results may swap order
- filters and thresholds can reduce final result count
- pagination can be less stable than relational pagination
- higher recall settings usually increase latency
- index updates can change nearby-neighbor paths
These behaviors should be understood and tested rather than treated as ordinary application bugs.
ANN Means Approximate
ANN stands for approximate nearest neighbor.
An exact nearest-neighbor search compares the query vector with every vector in the collection. That can be too slow for millions or billions of vectors.
An ANN index organizes vectors so the search can inspect only a useful part of the index. This makes retrieval fast, but it can miss some true nearest neighbors.
Recall Is a Trade-Off
Recall measures how many of the true nearest neighbors the search returns.
Higher recall usually requires more work at query time or build time. Lower recall can be faster but may miss some expected results.
This is the central ANN trade-off:
higher recall = more search work = usually higher latency
lower latency = less search work = possible recall lossTop-K May Not Match Brute Force Exactly
An ANN index may return a strong approximate top-k list without matching the exact brute-force top-k list.
For many production search applications, this is acceptable if the returned results are relevant. For evaluation, debugging, or compliance-sensitive retrieval, compare ANN results against exact search on a sample set.
Limit Can Affect Search Depth
In some ANN systems, the requested limit influences how deeply the index searches.
For example, if the system dynamically sizes its search work based on the requested result count, asking for more results can cause the index to explore more candidates. That can improve recall and may surface a result that was missed with a smaller limit.
This means:
top 10 from limit 10may not always match:
first 10 results from limit 100when approximate search depth changes.
Search Depth Parameters Matter
Many graph-based ANN indexes expose query-time search-depth parameters. In HNSW-style indexes, this is often called ef.
A higher search-depth value explores more candidates and can improve recall. It also tends to increase latency.
A lower value can make search faster, but it increases the chance that some true nearest neighbors are missed.
Build Parameters Matter Too
Some recall behavior is determined when the index is built.
For HNSW-style indexes, parameters such as graph connectivity and construction depth affect the quality of the graph. Stronger build settings can improve recall, but they may increase memory usage and indexing time.
If query-time tuning is not enough, the index may need stronger build-time settings.
Equal-Distance Results May Change Order
When two results have identical or nearly identical distances, their order may not be stable unless the system applies a deterministic tie-breaker.
For example:
A distance = 0.17664719
B distance = 0.17664719Both orders may be valid by distance:
[A, B]
[B, A]If your product requires stable ordering, add a secondary sort key after retrieval, such as document ID, timestamp, or another deterministic field.
Near Ties Can Flicker
Even when distances are not exactly equal, very small score differences can produce result flicker.
This can happen because of floating-point precision, shard merge timing, concurrent execution, or approximate candidate selection.
When score gaps are tiny, the practical relevance difference between adjacent results may also be tiny.
Filters Can Change Result Shape
Filtered vector search can behave differently from unfiltered vector search because only a subset of objects is eligible.
Depending on the system, filters may be applied before, during, or after candidate traversal. Restrictive filters can reduce result count, increase search work, or change which candidates are considered.
Always test important filters with realistic data distributions.
Thresholds Can Reduce Results
Distance or similarity thresholds can cause the final result set to contain fewer than k results.
This is usually desirable when the alternative is returning weak matches. But it means the application should handle short result lists and zero-result cases.
Pagination Can Be Tricky
Offset pagination is less natural for ANN search than for a relational table sorted by a stable column.
Approximate traversal, equal-distance ties, shard merging, filters, and index updates can all affect which items appear around page boundaries.
For user-facing search, prefer designs that do not require perfect deep pagination over approximate vector results.
Index Updates Can Change Results
ANN indexes are data structures. Adding, deleting, or updating vectors can change neighborhood relationships and traversal paths.
Even if the query does not change, results may shift after ingestion, re-embedding, re-chunking, or index maintenance.
This is normal in dynamic vector systems.
Compression Can Affect Recall
Quantization and compression reduce memory or storage cost, but they can also change candidate selection or scoring precision.
Many systems rescore top candidates with full-precision vectors to recover quality. If the candidate pool is too small, a true neighbor may not reach the rescoring stage.
When using compression, evaluate recall and latency together.
Distributed Search Can Affect Ordering
In distributed systems, results may be collected from multiple shards and merged.
If multiple results have the same or nearly the same distance, shard timing and tie-breaking behavior can affect final order.
This does not necessarily mean recall is wrong. It may only mean the ordering of equivalent candidates is not stable.
Hybrid Search Adds More Behavior
Hybrid search combines vector and keyword retrieval. That adds more sources of result movement:
- vector recall
- keyword candidate generation
- score fusion
- reranking
- filters
- limits on each candidate source
If hybrid results are surprising, inspect both retrieval paths and the fusion logic.
How to Debug ANN Result Behavior
Use this checklist:
- Compare ANN results with exact search on a sample.
- Increase query-time search depth and check whether recall improves.
- Check whether the requested limit changes internal search depth.
- Remove filters temporarily to isolate filter behavior.
- Remove thresholds temporarily to inspect raw candidates.
- Check whether result differences are only equal-distance ties.
- Log scores, IDs, metric, index type, and embedding model version.
- Test after ingestion or reindexing if the data recently changed.
How to Design Around ANN Behavior
For production systems:
- treat ANN results as relevance-ranked candidates, not a perfect total order
- use reranking when final precision matters
- calibrate thresholds on real queries
- avoid relying on deep offset pagination
- add deterministic tie-breaking where stable UI order matters
- monitor recall with benchmark queries
- separate retrieval quality issues from display ordering issues
When to Tune the Index
Tune ANN settings when:
- known relevant items are often missing
- results change too much with small limit changes
- RAG answers fail because key chunks are not retrieved
- recall benchmarks are below target
- filters consistently reduce candidate quality
Expect trade-offs. Higher recall often costs latency, memory, indexing time, or throughput.
Common Mistakes
Common mistakes include:
- expecting ANN to behave exactly like brute-force search
- assuming limit only truncates final results
- ignoring search-depth settings
- treating equal-distance order changes as relevance bugs
- using deep offset pagination as a reliability mechanism
- testing only unfiltered queries while deploying filtered search
- changing compression settings without recall evaluation
Summary
ANN indexes make vector search fast by trading some exactness for speed. As a result, search results can change with limit, search depth, filters, thresholds, compression, updates, and tie-breaking behavior.
The right response is not to avoid ANN indexes, but to understand their behavior, tune recall where needed, evaluate with real queries, and design application logic that handles approximate retrieval gracefully.