Add compressed vector storage (Int8 SQ, RaBitQ, TurboQUANT)

[sven-n]

Summary

Adds an IVectorEncoding abstraction so a vector database can store its embeddings in compressed form to save memory and disk space. The encoding is injected at the database constructor and the codec owns the asymmetric (query float vs. stored encoded) distance computation, so search no longer has to decode back to float[].

Four built-in encodings ship in this PR:

Id	Compression	Notes
raw-f32	1x (default)	Backwards-compatible passthrough
int8-sq	~4x	Symmetric per-vector int8 scalar quantization
rabitq	~32x	Rotation-free RaBitQ — 1-bit sign code + per-vector norm and correction factor
turboquant	~8x	4-bit symmetric SQ, nibble-packed on disk

Usage:

var db = new BasicMemoryVectorDatabase(Int8ScalarQuantizationEncoding.Instance);

The original parameter-less constructors remain (raw encoding, same behavior as before).

Backward compatibility

• Existing .b59vdb files load unchanged — the regression test (VectorDatabaseVersion_2_0_2_001) still passes with bit-exact similarity (0.3396831452846527).
• Raw-encoded databases write the same legacy JSON shape, byte-for-byte: VectorEncodingId is omitted from database.json when the encoding is raw, and the per-item JSON keeps the {"Vector": [...]} array. A JsonConverter on VectorTextItem<,> understands both the legacy shape and the new {"EncodingId", "Dimensions", "EncodedBytes"} shape.
• New nullable IEncodedVector EncodedVector { get; set; } on IVectorTextItem<,> has a default implementation that adapts to/from the existing float[] Vector, so external implementations keep compiling.

Key design choices (called out for review)

• Codec owns distance. IVectorComparer gained a VectorComparison MetricKind { get; } property so the database can dispatch to the encoding's fast asymmetric path: encoding.Compare(comparer.MetricKind, queryFloat, stored.EncodedVector). External IVectorComparer implementations will get a compile error pointing them at the new requirement; this is the only intentional public-API break.
• VectorTextItem.Vector becomes a computed view — getter decodes from EncodedVector, setter wraps in raw encoding. No values are silently cached.
• VectorEncodingRegistry holds singleton instances keyed by id so persisted vectors can be rehydrated to the right codec on load. The encoding actually used after a Load is whatever is recorded in the file, not the constructor parameter.
• RaBitQ caveat. The published algorithm pre-rotates database and query vectors through a shared random orthonormal matrix for stronger concentration bounds. That would require per-database state which the registry-singleton IVectorEncoding shape does not model, so the rotation is omitted here. For already-isotropic embedding outputs (the typical input) the rotation-free estimator is still close in practice. A rotation-based variant is a follow-up that would also touch the registry contract.

Test plan

• 23 new tests under SharpVectorTest/VectorEncoding/ cover: per-codec encode/decode round-trip, byte round-trip equivalence, cosine similarity accuracy bounds for random vectors, similarity-ranking correctness, exact storage-size expectations, odd-dimension nibble packing tail case, database-level save/load through each codec, and that raw-encoded database.json byte-omits the new VectorEncodingId field
• Full suite green except for two pre-existing flaky BasicDiskVocabularyStore / BasicDiskVectorStore WAL tests (AddAndGetText_PersistsToDisk, Delete_RemovesFromIndexButKeepsFile) that fail on main for an unrelated reason — those are fixed in Fix WAL file-handle leak in disk store recovery #89

🤖 Generated with Claude Code