Choosing the Right Index
No index is universally best. The right choice depends on your dataset size, latency requirements, recall requirements, and infrastructure budget. Here is a structured comparison:
| Feature | HNSW | IVF-Flat | IVF-PQ |
|---|---|---|---|
| Query Latency | Lowest (sub-ms) | Medium | Medium-Low |
| Recall | Highest (tunable via ef) |
High (exact within cells) | Medium (compression loss) |
| Memory Usage | High (vectors + graph) | High (full vectors) | Low (compressed vectors) |
| Training Required | No | Yes (K-Means) | Yes (K-Means + codebooks) |
| Incremental Inserts | Yes (no rebuild) | Partial | Partial |
| Best Dataset Size | < 100M vectors | < 10M vectors | 100M – 10B+ vectors |
| Best Use Case | Real-time AI, RAG, low-latency search | Moderate datasets, high recall | Massive scale, cost-constrained |
The Modern Hybrid: HNSW-PQ
The most sophisticated production deployments combine both approaches. HNSW-PQ (also called HNSW with scalar or product quantization) uses:
- The HNSW graph for navigation — its layered structure provides fast, high-recall traversal.
- PQ-compressed vectors stored at each graph node — instead of storing raw float32 vectors, each node stores its PQ code.
The search proceeds exactly like standard HNSW, but distance computations during traversal use ADC (fast table lookups) instead of exact dot products. At the final re-ranking step, the top candidates can optionally be re-scored using their original full-precision vectors (if stored separately) to recover precision.
This hybrid achieves:
- Memory: 10x–30x reduction vs. pure HNSW
- Recall: 5–10% lower than pure HNSW, but tunable
- Latency: Comparable to pure HNSW (ADC is fast)
Qdrant uses this approach with its HNSW + scalar quantization. Weaviate offers HNSW with PQ. Faiss (Meta's library) implements IVF-PQ as its flagship large-scale index.
Disk-Based Indexes: The Next Frontier
When even compressed vectors don't fit in RAM, disk-based ANN indexes move the vector data to NVMe SSDs while keeping only the graph structure or centroid index in memory. DiskANN (Microsoft Research) is the leading algorithm here. It achieves recall comparable to HNSW while storing vectors on disk, using a technique called beam search with SSD prefetching to hide I/O latency.
This matters for platform engineers because it changes the infrastructure equation: instead of provisioning expensive high-memory instances (e.g., AWS r7g.48xlarge with 1.5 TB RAM), you can use compute-optimized instances with large NVMe storage at a fraction of the cost. The trade-off is higher p99 latency due to disk I/O variance.
Senior depth — the index is not static: In production, vector indexes are not built once and forgotten. Data distributions shift (new products, new languages, new user behaviors). Centroids become stale. Graph quality degrades as vectors are deleted. Production systems need a strategy for index lifecycle management: scheduled full rebuilds, incremental segment merges (like Elasticsearch's segment architecture), and recall monitoring via periodic ground-truth sampling. A vector index with 95% recall at launch can silently degrade to 80% recall six months later without any alerts firing — because recall is not a metric most monitoring systems track by default.