perf(ggml-cpu): enable Q2_0 fast path on AVX-VNNI CPUs (3.2x on Intel 12th-14th gen)

[thtro]

Problem

ggml_vec_dot_q2_0_q8_0 (ggml/src/ggml-cpu/arch/x86/quants.c) gates its VNNI fast path on:

#if defined(__AVX512VNNI__) && defined(__AVX512VL__)

There is no AVX2/AVX-VNNI fallback, so every x86 CPU without AVX-512 silently takes the scalar loop.

This excludes all Intel consumer CPUs from 12th–14th gen (Alder Lake / Raptor Lake), where AVX-512 is fused off because of the P/E hybrid design — but which do have AVX-VNNI. That looks like a large share of the "on-device / runs on a standard laptop" x86 audience Bonsai targets. (AMD Zen 4/5 expose AVX512-VNNI and are unaffected, which may be why this went unnoticed.)

Symptom on such a host: prompt-eval throughput equals decode throughput, i.e. the per-token vec_dot dominates and batching gains nothing.

Fix

The fast-path body is already entirely 256-bit AVX2 — the only AVX-512 dependency is the _mm256_dpbusd_epi32 intrinsic. AVX-VNNI exposes the identical operation as _mm256_dpbusd_avx_epi32, so this aliases the intrinsic and widens the guard.

No algorithmic change, and no behavior change on AVX-512-VNNI hosts (same path, same intrinsic).

Results

Intel i5-13400 (Raptor Lake, 6P+4E, AVX-VNNI, no AVX-512), 12 threads, Debian 13, CPU-only build (-DGGML_NATIVE=ON). Same model, same prompt, temperature=0; the kernel is the only variable (controlled A/B: patch applied, reverted, rebuilt, re-applied).

Model	metric	before	after	speedup
Ternary-Bonsai-8B Q2_0	decode	2.17 tok/s	6.92 tok/s	3.2x
Ternary-Bonsai-8B Q2_0	prompt eval	2.7 tok/s	8.6 tok/s	3.2x

Repeat run after re-applying: 6.85 tok/s decode / 8.4 tok/s prompt (consistent).

With the patch, Ternary-Bonsai-4B Q2_0 reaches 12.4 tok/s decode / 16.2 tok/s prompt on the same host — which moves ternary Bonsai from "unusable" to "usable" on mainstream Intel desktops.

For reference, unpatched Ternary-Bonsai-27B Q2_0 runs at 0.69 tok/s on this host.

Verification

• echo | gcc -march=native -dM -E - on this host defines __AVXVNNI__=1 and __AVX2__=1, with no __AVX512VNNI__ — confirming the scalar path was taken before.
• vpdpbusd instruction count in libggml-cpu.so: 353 → 361 (the 8 added are the two Q2_0 call sites), confirming the fast path is compiled in.
• Generated output verified coherent before and after, on the identical prompt.

Notes / open questions

• The macro name GGML_DPBUSD_256 is arbitrary — happy to rename or switch to an inline helper if you prefer.
• Only compile-time dispatch is handled, matching the surrounding code's pattern. Builds using GGML_CPU_ALL_VARIANTS (runtime dispatch) may need an AVX-VNNI variant added to the variant list; I have not tested that path.
• I only tested Q2_0. Q1_0 may have the same gating issue but I have not verified it.

🤖 Generated with Claude Code

[thtro]

Some additional context after looking at the surrounding code and related PRs.

This aligns Q2_0 with the convention already in the same file. mul_sum_us8_pairs_float (~L105 of ggml/src/ggml-cpu/arch/x86/quants.c) already uses exactly this pattern:

static inline __m256 mul_sum_us8_pairs_float(const __m256i ax, const __m256i sy) {
#if defined(__AVX512VNNI__) && defined(__AVX512VL__)
    const __m256i summed_pairs = _mm256_dpbusd_epi32(zero, ax, sy);
    ...
#elif defined(__AVXVNNI__)
    const __m256i summed_pairs = _mm256_dpbusd_avx_epi32(zero, ax, sy);
    ...
#else

So _mm256_dpbusd_avx_epi32 under __AVXVNNI__ isn't a new idiom here — this PR just applies the existing one to ggml_vec_dot_q2_0_q8_0, which appears to be the only kernel in the file gated on AVX-512 exclusively. ggml_vec_dot_q1_0_q8_0 gates on plain #if defined(__AVX2__) and is unaffected.

That also explains the user-visible shape of the bug: on Alder/Raptor Lake, Q1_0 runs at full speed while Q2_0 silently drops to the scalar loop (~3x slower). Nothing warns you; it just looks like ternary is slow on CPU.

If the macro is unwelcome, an equally small alternative is a local static inline helper mirroring mul_sum_us8_pairs_float's structure — happy to switch.

Relation to #29: no overlap in scope — #29 (@pl752) implements bit-interleaved repack GEMM/GEMV for Q1_0, whereas this only touches the Q2_0 vec_dot guard. Worth noting they're complementary rather than alternatives: this PR does not add a batched path, so Q2_0 prefill throughput stays pinned to vec_dot speed (on my host prompt-eval ≈ decode, 8.6 vs 6.9 tok/s — the tell-tale sign of no GEMM). Repack kernels of the kind in #29, applied to Q2_0, look like the real fix for prefill; this is just the cheap unblock for the ISA gate in the meantime.

Note the two touch the same file (arch/x86/quants.c), though different functions, so there may be a trivial textual conflict depending on merge order — happy to rebase on whichever lands first.

[khosravipasha]

Thanks will review this more carefully.

We also have partial support in llama.cpp now (Q2_0). The Q2_0 there is a bit different and group size is 64 but for your changes should not matter. Feel free to send a PR to llama.cpp as well. We will probably pick this up on our side too after a bit of testimg.
ggml-org#24448

See for details of the migraion/upstream plan to llama.cpp for Q2_0:
https://github.com/PrismML-Eng/Bonsai-demo#upstream-status-for-ternary

[Copilot]

Pull request overview

Enables the Q2_0 VNNI fast path on AVX-VNNI CPUs without AVX-512.

Changes:

• Adds a portable 256-bit DPBUSD intrinsic alias.
• Extends the optimized path’s guard to AVX-VNNI.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[bri-prism]

Thanks, this is a well-made change. Verified on our side:

• The alias is correct and follows the existing convention in this file (mul_sum_us8_pairs_float uses the same AVX512-VNNI / __AVXVNNI__ split with the same pair of intrinsics). On AVX-512 hosts nothing changes; on AVX-VNNI hosts the new path is the same code the AVX-512 path has been running, VEX-encoded, so it inherits that path's validation.
• Your two open questions: (1) GGML_CPU_ALL_VARIANTS is already covered, the alderlake variant carries AVX_VNNI which defines __AVXVNNI__, so runtime-dispatch builds pick this up with no extra work. (2) Q1_0 is not affected, its x86 fast path is gated on plain __AVX2__ and uses maddubs rather than dpbusd, so there is no equivalent gap there.
• GGML_DPBUSD_256 is a fine name, no need to change it.

We can't reproduce the benchmark locally (no 12th-14th gen host in the loop right now) but the methodology and the instruction-count check are convincing, and the magnitude is consistent with what we see from VNNI paths elsewhere. Approving.