Local RAG and semantic search have a boring but expensive problem: embeddings are large.
A single float32 embedding costs four bytes per dimension. That sounds harmless until the corpus grows:
- 10 million vectors at 768 dimensions: about 30.72 GB for raw float32 values.
- 10 million vectors at 1,536 dimensions: about 61.44 GB for raw float32 values.
- 10 million vectors at 3,072 dimensions: about 122.88 GB for raw float32 values.
Those numbers are just the raw vector payload. A real system also stores document text, IDs, metadata, filters, persistence files, allocator overhead, caches, and whatever index structure the engine needs. That is why local vector search often turns into either a managed vector database bill or a bigger VM than the feature seemed to deserve.
The interesting part is that the raw vector payload does not always need to stay float32.
What TurboQuant Actually Claims
TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate is a 2025 paper by Amir Zandieh, Majid Daliri, Majid Hadian, and Vahab Mirrokni. The paper proposes a data-oblivious vector quantization method: instead of training a codebook on your corpus, it uses random rotation and scalar quantizers derived from the expected coordinate distribution of high-dimensional vectors.
That detail matters for application teams. A trained product-quantization setup can work very well, but it introduces an operational step: sample data, train codebooks, and think carefully about what happens when data distribution changes. TurboQuant's promise is different: online insertion without a separate train phase, while still getting near-optimal distortion within a small constant factor.
The paper covers both vector quantization theory and experiments. It reports quality-neutral KV-cache behavior around 3.5 bits per channel, marginal degradation around 2.5 bits per channel, and nearest-neighbor search results that beat the product-quantization baselines used in the paper. Those are strong results, but they are not a universal claim that every workload becomes lossless at every compression level.
Where TurboVec Fits
TurboVec is an open-source Rust vector index with Python bindings that implements TurboQuant-style compression for vector search. Its PyPI package is currently published as turbovec 1.0.0, but the package still carries the classifier Development Status :: 3 - Alpha.
That combination should guide how you adopt it:
- Treat the public API as real enough to prototype against.
- Treat production use as something that requires your own benchmark, recovery test, and upgrade plan.
- Do not assume that a
1.0.0package version cancels the explicit alpha classifier.
The verified Python API uses TurboQuantIndex and IdMapIndex, not a generic turbovec.Index class. The repository documents 2-bit and 4-bit compression, persistence through write, load, and sync, filtered search with ID allowlists, Rust APIs, Python APIs, and benchmark JSON files.
The Memory Math Without Hand Waving
Compression depends on bit width.
For a 1,536-dimensional vector:
- Float32: 1,536 dimensions x 4 bytes = 6,144 bytes per vector.
- 4-bit quantization: 1,536 dimensions x 4 bits = 768 bytes per vector before side data.
- 2-bit quantization: 1,536 dimensions x 2 bits = 384 bytes per vector before side data.
That is the clean part of the math. A 4-bit representation is an 8x reduction of the raw float32 coordinate payload. A 2-bit representation is a 16x reduction of that same payload.
The original version of this article mixed those cases. It said a 1,536-dimensional embedding costs 6,144 bytes and compresses to 384 bytes, which is the 2-bit case, while also describing the result as an 8x reduction, which is the 4-bit case. The corrected rule is simple:
- Use 4-bit when you are targeting the common 8x raw-vector reduction.
- Use 2-bit when you are willing to test a more aggressive 16x raw-vector reduction.
Neither number includes metadata, deleted-slot behavior, tenant maps, persisted files, document text, or the memory your application uses around the index.
What the Public Benchmarks Show
TurboVec's repository publishes benchmark results for 100K vectors, 1K queries, and k=64, including 1,536-dimensional and 3,072-dimensional OpenAI-style embeddings plus 200-dimensional GloVe embeddings. The project compares against FAISS IndexPQ and IndexPQFastScan configurations sized to similar bit rates.
The headline is encouraging, especially for high-dimensional embeddings: TurboVec reports strong recall at 1,536 and 3,072 dimensions, direct SIMD search over compressed vectors, and better insertion and removal characteristics than trained FAISS PQ in the published benchmark cells.
The caveat is just as important: those are project-published benchmarks at 100K vectors, not independent proof that every 10-million-document workload will fit in a specific RAM budget or match the same latency profile. The repository's 10-million, 31 GB to 4 GB statement is based on the 768-dimensional raw-vector math and the project's compression assumptions. It is a useful sizing example, not a substitute for measuring your own corpus.
Why This Still Matters
Even with those caveats, the architecture impact is real.
If your use case is local semantic search, internal RAG, or privacy-sensitive retrieval, cutting the raw vector payload by 8x can change the deployment conversation. A feature that previously required a managed vector service or a memory-heavy local database may become viable on ordinary infrastructure.
But the responsible decision is not "TurboVec means no more vector database." The responsible decision is:
- Benchmark the exact embedding model and dimensions you use.
- Measure recall at the
kyour product actually serves. - Include metadata and document storage in the memory budget.
- Test persistence, reload time, deletion behavior, and crash recovery.
- Decide whether an alpha-classified package fits your operational risk.
For some teams, a mature vector database remains the right answer. For others, a local compressed index behind a clear application boundary is exactly the right trade.
The Practical Takeaway
TurboQuant is valuable because it attacks the real bottleneck: vectors are too large to keep treating float32 as the default forever. TurboVec is valuable because it makes that research approachable through Rust and Python APIs.
The corrected claim is narrower than the old headline, but more useful: modern quantization can reduce the raw vector payload by 8x or 16x, and TurboVec provides a credible implementation to test. Whether that makes your system small enough depends on dimensions, bit width, metadata, index overhead, and the recall your users need.
That is not a magic escape hatch from infrastructure planning. It is better: it is an engineering knob you can measure.