Vector compression — making an index affordable

This is the first of three cards I added after checking the official skills outline against what the material I had actually taught. The video course this was built from is a developer course: it teaches you to query an index and never to tune, secure or operate one. Three cards close that seam. All of it was verified against Microsoft's own documentation on 5 August 2026.

Vectors are expensive to store. Compression makes a large index affordable, and it costs you some accuracy — which you then buy back.

The three levers

Scalar quantization, sometimes written SQ8, converts each number from a 32-bit float to an 8-bit integer. Eight bits gives 256 possible values, so each component is placed in one of 256 bins. The vector index comes out around four times smaller — roughly 75% saved. This is the safe default.

Binary quantization reduces each component to a single bit. Savings reach 96%, around 28 times smaller. It works best above 1024 dimensions and with zero-centred embeddings, which is what OpenAI, Cohere and Mistral produce.

"stored": false drops the retrievable raw copy, saving up to 50% more. It is irreversible, can only be set at index creation, and forces retrievable: false.

Compression is lossy, so oversample

1. Query compressed — the search runs over the quantized vectors.
2. Oversample — fetch k times more candidates than you need. With k=5 and an oversampling factor of 20, that is 100 candidates.
3. Rescore — score those candidates against full-precision vectors, or against the binary dot product.
4. Return top k — only k comes back. Five, not a hundred.

The logic is worth holding: compression makes the shortlist slightly unreliable, so you take a longer shortlist and then rank it accurately. You pay a little more retrieval work to recover the accuracy compression cost you.

The scalar and binary asymmetry

• Scalar rescoring needs the full-precision originals.
• Binary can rescore on the dot product of the binary embedding instead.

Which means discardOriginals is sensible for binary and destructive for scalar — discard the originals under scalar quantization and you have removed the thing rescoring depends on. Both settings are irreversible.

Properties to recognise

• vectorSearch.compressions — defines a new profile, applied to a new field
• quantizedDataType — int8 only
• defaultOversampling — defaults to 4
• rescoreStorageMethod — preserveOriginals (the default) or discardOriginals
• truncationDimension — Matryoshka representation learning, for text-embedding-3

The counterintuitive part

With preserveOriginals, quantization cuts memory but slightly raises total storage — because both copies are held. If you measured success by storage consumed, compression appears to have made things worse.

With discardOriginals, storage strictly falls.

And query-time oversampling overrides the index setting, forcing rescoring on.

The trap

Rescoring works on HNSW only.

Exhaustive nearest-neighbour search already scans every vector, so there is no shortlist to widen and nothing to rescore. Oversampling is meaningless there, and unsupported. Any answer combining exhaustive KNN with oversampling is wrong.