turbovec: shrink the RAM a RAG index needs

RAG turns documents into embeddings and stores them. As float32³, ten million 768-dimensional vectors take 31 GB, which fills RAM before any model is loaded.

turbovec stores those vectors at 2 or 4 bits per dimension and searches the compressed form, so the same ten million fit in 4 GB.

It is a library, not a service: it runs inside your process and persists to one file, so it slots into an existing pipeline⁴.

float32 is 4 bytes per dimension, 4-bit is 0.5 and 2-bit is 0.25, so the arithmetic gives 8 and 16 times. The README reports measured factors of 7.8 and 15.5.

At 100K vectors of 1536 dimensions, float32 takes 586 MB, 4-bit 75 MB and 2-bit 38 MB. Budgeting with that ratio avoids running out of RAM.

Install with pip and pass a NumPy array. Vectors are searchable as soon as they are added, with no training step.

Input must be a 2-D float32 array. Other dtypes raise instead of converting, which catches conversion mistakes early.

The scores are inner products, so normalise vectors before adding if you want cosine similarity.

import numpy as np
from turbovec import TurboQuantIndex

vectors = np.load("embeddings.npy").astype(np.float32)  # (n, 1536)
index = TurboQuantIndex(dim=1536, bit_width=4)
index.add(vectors)

scores, slots = index.search(vectors[:1], k=5)
index.sync("index.tv")

4-bit search runs 3.8 to 5.1 times faster than FAISS: 4.0 times on ARM and 4.9 on x86, so both architectures gain.

Recall@1 is 0.959 and 0.981 at 4-bit against FAISS at 0.966 and 0.971, for 1536 and 3072 dimensions.

2-bit drops to 0.901 and 0.931, and low dimensions lose more: GloVe at 200 dimensions reaches 0.572. That is the number that decides the width.

Measured on 100K embeddings, k=64, 1,000 queries, median of 5 runs, on GCP c4a-standard-8 (ARM) and c3-standard-8 (x86).

With no training step, an added vector is searchable at once, so a pipeline that keeps ingesting documents never has to reindex.

Deletes take microseconds. FAISS repacks codes on every delete, which runs 0.19 to 1.03 seconds.

Saving writes only the change, so it stays in milliseconds however large the index, and a crash keeps everything up to the last sync.

Pass an allowlist⁵ and the search only returns from it, which lets permissions or a time window live outside the index.

IdMapIndex keeps your own ids instead of slot positions, which makes matching keys from another system straightforward.

from turbovec import IdMapIndex

index = IdMapIndex(dim=1536, bit_width=4)
index.add_with_ids(vectors, ids)

allowed = np.array([1001, 1003, 1042], dtype=np.uint64)
scores, ids = index.search(query, k=10, allowlist=allowed)

Store integrations exist for LangChain, LlamaIndex, Haystack and Agno, matching each framework's in-memory store so the swap is a replacement.

With a local embedding model the whole index runs offline, and turbovec itself never leaves the process.

It is not recommended when you need a graph index: it scans every vector, so cost grows linearly. At 100K a query is 0.13 to 0.25 ms on eight cores, and for hundreds of millions behind strict latency, HNSW or IVF is the right shape.

Exact float results are out of reach: quantization⁶ is lossy, and at 4-bit the top hit matches the float answer 96 to 98 percent of the time. Check recall on a sample of your own data.

It is not a server either: no network API⁷, replication or multi-tenancy, it is a component you place inside a stack that has those.

Dimensions must be multiples of 8 and no wider than 16,384, and the only widths are 2 and 4 bits.