ggml-cpu: enable Q2_0 VNNI kernel on AVX-VNNI-only CPUs#76
Open
gondoi wants to merge 1 commit into
Open
Conversation
The fast path in ggml_vec_dot_q2_0_q8_0 was gated on __AVX512VNNI__ && __AVX512VL__, which is stricter than needed: the kernel body uses only 256-bit registers, so it runs unchanged on CPUs that have AVX-VNNI but no AVX512 (e.g. Intel Alder Lake / Raptor Lake). The only difference is the intrinsic name: _mm256_dpbusd_avx_epi32 instead of _mm256_dpbusd_epi32. Extend the gate with defined(__AVXVNNI__) and select the intrinsic via a local macro. Builds with -DGGML_AVX_VNNI=ON (or -march=native on supporting CPUs) now take the vectorized path. Measured on i7-12650H (Raptor/Alder Lake, no AVX512) with Ternary-Bonsai-27B Q2_g64: ~8x decode speedup vs the scalar fallback, approaching the memory-bandwidth limit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Q2_0 ternary fast path in
ggml_vec_dot_q2_0_q8_0(ggml/src/ggml-cpu/arch/x86/quants.c) is gated on#if defined(__AVX512VNNI__) && defined(__AVX512VL__). That gate is stricter than the code requires: CPUs with AVX-VNNI but no AVX512 — notably Intel Alder Lake and Raptor Lake client parts, where AVX512 is fused off — fall through to the scalar loop and leave most of the available throughput on the table.Change
Extend the gate to
(defined(__AVX512VNNI__) && defined(__AVX512VL__)) || defined(__AVXVNNI__)and select the dot-product intrinsic via a local macro:_mm256_dpbusd_epi32(unchanged)_mm256_dpbusd_avx_epi32Why this is safe
__m256i) and 128-bit registers — no 512-bit state, no masking, nothing AVX512-specific.vpdpbusdoperation on ymm registers; the intrinsic name (and VEX vs EVEX encoding) is the only difference. Numeric behavior is bit-identical.Measured results
On an i7-12650H (Alder Lake, AVX-VNNI, no AVX512) running Ternary-Bonsai-27B Q2_g64 (7.6 GB): stock scalar path decodes at 0.54 tok/s; with this patch the vectorized path is enabled, giving ~8x decode speedup, approaching the memory-bandwidth limit.
Enabling it
The build system already supports this:
-DGGML_AVX_VNNI=ON(which defines__AVXVNNI__, seeggml/src/ggml-cpu/CMakeLists.txt) or the default-march=nativeon a supporting CPU enables the path automatically.Compile checks
Both configurations build cleanly (gcc 15.2,
-DGGML_NATIVE=OFF -DGGML_AVX=ON -DGGML_AVX2=ON -DGGML_FMA=ON -DGGML_F16C=ON, with and without-DGGML_AVX_VNNI=ON). Disassembly confirmsvpdpbusdis emitted inggml_vec_dot_q2_0_q8_0for the VNNI build and absent (scalar fallback) in the default build.🤖 Generated with Claude Code