feat: SVAR2 read-bound dataset wiring (draft — blocked on genoray svar-2 release)

[d-laub]

SVAR2 read-bound dataset wiring

Wires genoray's .svar2 sparse variant format into gvl.write and the gvl.Dataset read path. .svar2 is now a supported write variant source (a write-time ranges cache) and a live Dataset read backend that reconstructs all four output modes (reference, haplotypes, variants, tracks) via one all-Rust FFI per read — gathering off the cache with no interval-search-tree rebuild and no dense-union rebuild, byte-identical to the .svar/union-oracle path. It additionally supports the variant-windows sequence mode and unphased_union ploidy folding (see "Latest addition" below).

⚠️ DRAFT — blocked on the genoray release gate. This branch dev-wires genoray's svar-2 branch (@ b79e83a) via local path-deps + a local wheel:

• Cargo.toml: genoray_core = { path = "/carter/.../genoray", package = "genoray", default-features = false } and svar2-codec = { path = "/carter/.../genoray/svar2-codec" }
• pixi.toml: genoray = { path = ".../target/wheels/genoray-2.15.0-...whl" }

Upstream CI will not build until genoray svar-2 is published and these are flipped to the released crates.io/PyPI versions. Merge only after: (1) genoray svar-2 released (cz bump + publish), (2) path-deps → published versions, (3) pyo3/numpy pins reconciled with the release. This PR captures the complete, reviewed implementation for that moment.

What's included

The full svar2-m6b-kernel feature branch (69 commits ahead of main): the earlier M6b two-source union-path kernels, the read-bound wiring, the read-bound getitem perf optimizations (B1–B5), and — most recently — the variant-windows + unphased_union read path. Core pieces:

• Write — Svar2Link/Svar2Fingerprint + Metadata.svar2_link; _write_from_svar2 streams a 6-array ranges cache under genotypes/svar2_ranges/ (sized to the dataset's selected samples); .svar2 write dispatch.
• Rust read core — links genoray's query-only genoray_core crate + Svar2Store pyclass (one ContigReader per contig, opened once). Read-bound kernels reuse the already-validated union kernels by marshalling BatchResultSplit into their flat layout via svar2::split_to_flat (no forked kernels): reconstruct_haplotypes_from_svar2_readbound, shift_and_realign_tracks_from_svar2_readbound, decode_variants_from_svar2_readbound, hap_diffs_from_svar2_readbound.
• Dataset dispatch — a separate Svar2Haps reconstructor (SVAR1 Haps byte-unchanged) for haplotypes + variants; an additive HapsTracks branch for realigned tracks; Dataset.open(svar2=<override>) mirroring svar=. The live union (overlap_batch) dispatch is retired to a test-only parity oracle.
• Read-bound getitem perf — byte-identical read-path optimizations that brought haplotypes (the primary training-read mode) to parity with .svar (from 12–25× slower). The wins, in order of impact:
  • Single-group fast-path (gvl 63b8b87) — cProfile showed ~96% of the svar2 haplotype call was a numpy arange+repeat ragged-row permutation to un-group per-contig kernel output back to global order; for a single-contig batch that permutation is the identity. Skipping it dropped haplotypes 12–25× → ~1.1–1.2×.
  • genoray dense-presence block-copy (genoray b79e83a, 3ff8942) — replaced gather_haps_readbound's bit-by-bit per-hap presence loop with a per-query threshold + byte-batched copy_bits (self-time 8.85% → 3.31%).
  • Variant-path fusion (gvl b84a57d) — decode_variants_from_split decodes straight from the gather via a streaming 3-way merge, dropping the split_to_flat marshaling copy (variants germline 3.0–3.2× → 2.6×, somatic → 3.66×).
  • Plus the earlier B1–B5 instruction-level reductions (B4a genoray gather_haps_readbound asm ≈−8.5% variants_germline; B1 −6.3% haplotypes). All byte-identical; svar2 parity suite held green throughout. Consolidated in tmp/svar2_mvp/prof_out/bench_gvl/OPTIMIZATION.md + readbound/RESULTS.md.
• Requires a pyo3 0.28→0.29 + numpy 0.28→0.29 bump (to match genoray svar-2); validated additive (full Rust suite green).

Latest addition — variant-windows + unphased_union (pure-Python composition, no Rust change)

Dataset.open(..., svar2=...).with_seqs("variant-windows", VarWindowOpt(...)) (previously NotImplementedError) now works, and unphased_union is enabled for both svar2 decode modes (variants and variant-windows). Implemented by composing the already-validated svar2 variant decode with the already-validated window-assembly Rust kernel (assemble_variant_buffers) via an identity gather, per contig group — no new/changed Rust.

• ref="window" with alt ∈ {window, allele}. ref_window is byte-identical to the SVAR1 backend; alt/alt_window differ only for pure-deletion records (the empty-vs-anchor ALT convention below), validated instead via svar2-internal decomposition + tokenized variants.alt.
• unphased_union folds ploidy 2→1 (order-preserving, no dedup) — the same row_offsets[::P], eff_ploidy=1 fold SVAR1 uses; start/ilen byte-identical to the SVAR1 union. When the flag is off, both reconstructors reduce byte-for-byte to the pre-existing diploid path.
• Still guarded in this mode: ref="allele" (needs stored REF bytes .svar2 lacks → ValueError), and max_jitter>0/jitter>0 (read-bound decode has no right-clip → NotImplementedError).
• Docs updated (skills/genvarloader/SKILL.md, docs/source/{faq,format}.md); perf/fusion for this mode is deferred (see follow-ups).

Parity & verification

• All four modes byte-identical to the .svar/union oracle (SparseVar2Source, genoray decode). SVAR1 path is additive (byte-unchanged). 49 svar2-selected tests across the tree (21 in tests/dataset/test_svar2_dataset.py, incl. 10 variant-windows/unphased_union cases); full tree 1040 passed / 0 failed / 0 errors.
• Two documented, intentional non-identities: (1) a pure deletion's variants/variant-windows ALT is empty b"" for .svar2 vs the VCF anchor base for .svar (genoray atomized convention; haplotypes and ref_window unaffected); (2) SVAR1's max_ends same-POS-tie under-extension is a latent SVAR1 bug (docs/known-issues/svar1-max-ends-tie-underextension.md) and those ties are excluded from parity.
• Now supported (previously guarded): variant-windows (ref="window", alt ∈ {window, allele}) and unphased_union (for variants and variant-windows).
• Still guarded — splice, exonic/keep, min_af/max_af, annotated, in-kernel RC, ref="allele" for variant-windows, fixed-length tracks, variants/variant-windows + max_jitter>0/jitter>0, and multi-contig FlankSample all raise NotImplementedError/ValueError (never silently mis-compute).

Benchmark (chr21, matched .svar/.svar2, warm Dataset[:, :] through the full gvl getitem path)

Current numbers, after the read-parity optimization effort (single-group fast-path + genoray block-copy + variant fusion above). Both backends go through the full gvl Dataset.__getitem__ path (svar1 via variants=<p>.svar, svar2 via variants=SparseVar2(<p>.svar2)), interleaved per-repeat so shared-node drift hits both equally — the ratios are trustworthy; absolute ms are node-dependent. Workload: chr21, 20×2 kb windows, ALL samples, max_jitter=0. Harness: tmp/svar2_mvp/bench_gvl_svar1_vs_svar2.py; consolidated in tmp/svar2_mvp/prof_out/bench_gvl/OPTIMIZATION.md.

mode	germline (n=3202)	somatic (n=16007)
haplotypes	.svar1 1.08–1.11× faster → PARITY	.svar1 1.15–1.21× faster → PARITY
variants	.svar1 2.6× faster	.svar1 3.66× faster
store size	.svar2 5.7× smaller (193 MiB vs 1.07 GiB)	.svar2 1.45× smaller (36 vs 53 MiB)

(Starting point before this effort was haplotypes 12–25× slower; the single-group fast-path — killing the wasted numpy reorder — plus the genoray dense-presence block-copy closed it to parity.)

• Haplotypes — the primary training-read mode — is at parity. svar2 reconstructs haplotypes with the same live per-sample gather + codec decode that svar1 does after its own reconstruct, so once the numpy-glue overhead was removed the two converge.
• Variants stays ~2.6–3.7× and is architectural, not an unoptimized path. svar1's variant read is trivially cheap — it returns pre-materialized sparse int32 variant indices; svar2 must run the full per-hap gather + codec decode (pos/ilen/alt) live off the compact store, ~half gather / ~half decode with no single dominant culprit. Closing it would require either materializing a lighter per-sample variant index at write time (giving up part of svar2's storage advantage — its whole point) or a deeper gather→decode fusion (incremental, unlikely to reach full parity alone). This is a storage-vs-read-speed design decision, tracked as a follow-up.
• DSO-split structural win (met): perf of the warm .svar2 read loop shows zero SearchTree::build, dense_union, or overlap_batch self-time — the whole-cohort union/oracle path is absent; the read-bound path rebuilds no per-read interval tree or dense union.

Bottom line: .svar2 now matches .svar on haplotype read latency while being up to 5.7× smaller on disk (dense germline cohorts). Variants read remains ~2.6–3.7× slower — the cost of decoding the compact codec live rather than pre-materializing indices — which is the intended storage-vs-speed tradeoff. (Benchmarks predate the variant-windows/unphased_union addition, which adds a new mode and does not alter the variants/haplotypes read-path perf measured here.)

Follow-ups (out of scope; guarded)

• Variants read-latency parity is an architectural open decision (~2.6–3.7× slower). Options: (A) materialize a lighter per-sample variant index at write time — trades away part of svar2's storage advantage; (B) deeper gather→decode fusion for the variants path — incremental, unlikely to reach full parity alone. Haplotype parity is already achieved. Profiled-but-deferred asm targets on the residual: genoray decode_key (2.33%), merge_keys (2.31%), SearchTree::build (1.54%).
• Perf/fusion for variant-windows + unphased_union (the correctness path landed; the fused-decode perf pass is deferred).
• Variants/variant-windows right-clip for max_jitter>0/jitter>0 (to lift that guard); ref="allele" for variant-windows (needs stored REF bytes).
• Global-query-index remap to lift the multi-contig FlankSample guard.
• .pyi stubs for the svar2 FFI (pre-existing bigwig-only gap).

🤖 Generated with Claude Code