Skip to content

Avoid redundant LUT buffer initialization#29690

Open
GopalakrishnanN wants to merge 1 commit into
microsoft:mainfrom
GopalakrishnanN:gnallasamy-microsoft-benchmark-overwrite-allocation
Open

Avoid redundant LUT buffer initialization#29690
GopalakrishnanN wants to merge 1 commit into
microsoft:mainfrom
GopalakrishnanN:gnallasamy-microsoft-benchmark-overwrite-allocation

Conversation

@GopalakrishnanN

Copy link
Copy Markdown
Contributor

Description

  • Allocate the temporary MLAS LUT buffer with std::make_unique_for_overwrite.
  • Keep the existing full-buffer memset, preserving zero-initialized LUT data and padding while avoiding the allocation's redundant value-initialization pass.

Motivation and Context

MlasLutGemm currently value-initializes the temporary byte buffer and immediately zeroes the entire buffer again. The buffer ranges from a few KB during decode to multiple MB for representative prefill shapes, so the first write consumes memory bandwidth without affecting behavior.

Local Windows MSVC Release measurements used four alternating baseline/changed process runs with five repetitions each. The unaffected Dequant+SGEMM fallback served as a control.

Shape (M, N, K) Before After Raw change Control-normalized change
(128, 384, 1024) 478.3 us 454.3 us -5.0% -4.7%
(128, 1024, 4096) 3.506 ms 3.367 ms -4.0% -4.4%

Testing

  • Built onnxruntime_mlas_benchmark and onnxruntime_mlas_test in Release mode.
  • onnxruntime_mlas_test.exe --gtest_filter=SQLutGemmBlkBitWidth2BlkLen*.* (102 tests passed).
  • lintrunner -a onnxruntime/core/mlas/lib/qlutgemm.cpp.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes the MLAS MlasLutGemm temporary LUT buffer allocation by avoiding redundant value-initialization while preserving the existing full-buffer zeroing that the implementation relies on.

Changes:

  • Switch LUT scratch buffer allocation to std::make_unique_for_overwrite<int8_t[]> to skip default initialization.
  • Keep the existing memset(..., 0, lut_buffer_size) to preserve the required zero-initialized LUT data/padding.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants