Skip to content

NOT TO BE MERGED: Python script to benchmark mxr files - consumes mxr files with json encoding#4983

Draft
ahsan-ca wants to merge 1 commit into
benchmark-mxr-format-changefrom
benchmark-script-python
Draft

NOT TO BE MERGED: Python script to benchmark mxr files - consumes mxr files with json encoding#4983
ahsan-ca wants to merge 1 commit into
benchmark-mxr-format-changefrom
benchmark-script-python

Conversation

@ahsan-ca

Copy link
Copy Markdown
Contributor

Motivation

For REVIEW purposes only.

Technical Details

Changelog Category

Add a CHANGELOG.md entry for any option other than Not Applicable

    • Added: New functionality.
    • Changed: Changes to existing functionality.
    • Removed: Functionality or support that has been removed. (Compared to a previous release)
    • Optimized: Component performance that has been optimized or improved.
    • Resolved Issues: Known issues from a previous version that have been resolved.
    • Not Applicable: This PR is not to be included in the changelog.

@ahsan-ca ahsan-ca requested review from kahmed10 and pfultz2 June 17, 2026 18:42
@ahsan-ca ahsan-ca self-assigned this Jun 17, 2026
@ahsan-ca ahsan-ca requested a review from causten as a code owner June 17, 2026 18:42
Copilot AI review requested due to automatic review settings June 17, 2026 18:42
@ahsan-ca ahsan-ca marked this pull request as draft June 17, 2026 18:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new developer utility script under tools/ intended to benchmark GPU-dumped .mxr artifacts and produce a problem_cache.json consumable via MIGRAPHX_PROBLEM_CACHE.

Changes:

  • Introduces tools/benchmark_mxr.py to load .mxr files, run warmup/benchmark iterations, and emit a consolidated problem cache JSON.

Comment thread tools/benchmark_mxr.py
Comment on lines +1 to +10
#!/usr/bin/env python3
"""Benchmark dumped MIGraphX .mxr files and emit a problem cache.

.mxr files dumped via MIGRAPHX_GPU_DUMP_BENCHMARK_MXR.

After problem_cache.json file is generated by this script,
use MIGRAPHX_PROBLEM_CACHE=problem_cache.json to consume it.

Usage: benchmark_mxr.py <mxr_dir>
"""
Comment thread tools/benchmark_mxr.py
Comment thread tools/benchmark_mxr.py
@codecov

codecov Bot commented Jun 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Additional details and impacted files
@@                     Coverage Diff                      @@
##           benchmark-mxr-format-change    #4983   +/-   ##
============================================================
  Coverage                        92.73%   92.73%           
============================================================
  Files                              594      594           
  Lines                            31340    31340           
============================================================
  Hits                             29063    29063           
  Misses                            2277     2277           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@gh-app-migraphx-bot-pr-write

Copy link
Copy Markdown
Test Batch New Rate (31e77e) Old Rate (82079f) Diff Status
torchvision-resnet50 64 nan 2,825.64 nan
torchvision-resnet50_fp16 64 nan 6,481.47 nan
torchvision-densenet121 32 nan 348.79 nan
torchvision-densenet121_fp16 32 nan 1,628.64 nan
torchvision-inceptionv3 32 nan 301.44 nan
torchvision-inceptionv3_fp16 32 nan 881.79 nan
cadene-inceptionv4 16 nan 82.53 nan
cadene-resnext64x4 16 nan 105.15 nan
slim-mobilenet 64 nan 3,946.77 nan
slim-nasnetalarge 64 nan 187.41 nan
slim-resnet50v2 64 nan 1,208.71 nan
bert-mrpc-onnx 8 nan 468.18 nan
bert-mrpc-tf 1 nan 483.76 nan
pytorch-examples-wlang-gru 1 nan 323.66 nan
pytorch-examples-wlang-lstm 1 nan 548.03 nan
torchvision-resnet50_1 1 nan 754.98 nan
cadene-dpn92_1 1 nan 446.57 nan
cadene-resnext101_1 1 nan 363.17 nan
onnx-taau-downsample 1 nan 399.25 nan
dlrm-criteoterabyte 1 nan 31.34 nan
dlrm-criteoterabyte_fp16 1 nan 47.93 nan
agentmodel 1 nan 9,953.32 nan
unet_fp16 2 nan 13.15 nan
resnet50v1_fp16 1 nan 149.91 nan
resnet50v1_int8 1 nan 926.35 nan
bert_base_cased_fp16 64 nan 704.66 nan
bert_large_uncased_fp16 32 nan 341.37 nan
bert_large_fp16 1 nan 205.52 nan
distilgpt2_fp16 16 nan 2,089.95 nan
yolov5s 1 nan 564.52 nan
tinyllama 1 nan 36.21 nan
vicuna-fastchat 1 nan 34.08 nan
whisper-tiny-encoder 1 nan 86.70 nan
whisper-tiny-decoder 1 nan 417.85 nan
llama2_7b 1 nan 20.29 nan
qwen1.5-7b 1 nan 13.68 nan
phi3-3.8b 1 nan 26.63 nan
llama3-8b 1 nan 21.77 nan
whisper-large-encoder 1 nan 7.43 nan
whisper-large-decoder 1 nan 105.98 nan
mistral-7b 1 nan 23.81 nan
FLUX.1-schnell 1 nan 765.15 nan

Regressions detected 🔴

@gh-app-migraphx-bot-pr-write

Copy link
Copy Markdown
Test Status Result
bert-mrpc-onnx PASSED: MIGraphX meets tolerance
bert-mrpc-tf ERROR - check error output
traceback
Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 377, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 313, in main
import tensorflow as tf
File "/usr/local/lib/python3.10/dist-packages/tensorflow/init.py", line 38, in
from tensorflow.python.tools import module_util as _module_util
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/init.py", line 36, in
from tensorflow.python import pywrap_tensorflow as _pywrap_tensorflow
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 26, in
self_check.preload_check()
File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/platform/self_check.py", line 63, in preload_check
from tensorflow.python.platform import _pywrap_cpu_feature_guard
ImportError: libamdhip64.so.6: cannot open shared object file: No such file or directory
pytorch-examples-wlang-gru PASSED: MIGraphX meets tolerance
pytorch-examples-wlang-lstm PASSED: MIGraphX meets tolerance
dlrm-criteoterabyte PASSED: MIGraphX meets tolerance
agentmodel PASSED: MIGraphX meets tolerance
unet PASSED: MIGraphX meets tolerance
resnet50v1 PASSED: MIGraphX meets tolerance
bert_base_cased_fp16 PASSED: MIGraphX meets tolerance
bert_large_uncased_fp16 🔴 FAILED: MIGraphX is not within tolerance - check verbose output
bert_large PASSED: MIGraphX meets tolerance
yolov5s PASSED: MIGraphX meets tolerance
tinyllama PASSED: MIGraphX meets tolerance
vicuna-fastchat PASSED: MIGraphX meets tolerance
whisper-tiny-encoder PASSED: MIGraphX meets tolerance
whisper-tiny-decoder PASSED: MIGraphX meets tolerance
distilgpt2_fp16 PASSED: MIGraphX meets tolerance
llama2_7b PASSED: MIGraphX meets tolerance
qwen1.5-7b PASSED: MIGraphX meets tolerance
phi3-3.8b PASSED: MIGraphX meets tolerance
llama3-8b PASSED: MIGraphX meets tolerance
whisper-large-encoder ERROR - check error output
traceback
Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 377, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 224, in main
model = migraphx.parse_onnx(model_name, default_dim_value=batch)
RuntimeError: /data/src/include/migraphx/op/convolution.hpp:102: normalize_compute_shape: CONVOLUTION: mismatched channel numbers
whisper-large-decoder PASSED: MIGraphX meets tolerance
mistral-7b PASSED: MIGraphX meets tolerance
FLUX.1-schnell PASSED: MIGraphX meets tolerance

@ahsan-ca ahsan-ca force-pushed the benchmark-script-python branch from 31e77e7 to b84b23a Compare June 17, 2026 21:47
@ahsan-ca ahsan-ca force-pushed the benchmark-script-python branch from b84b23a to 91fd4b3 Compare June 17, 2026 21:49
@ahsan-ca ahsan-ca changed the base branch from develop to benchmark-mxr-format-change June 17, 2026 21:50
@ahsan-ca ahsan-ca requested a review from Copilot June 17, 2026 21:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

Comment thread tools/benchmark_mxr.py
Comment on lines +1 to +10
#!/usr/bin/env python3
"""Benchmark dumped MIGraphX .mxr files and emit a problem cache.

.mxr files dumped via MIGRAPHX_GPU_DUMP_BENCHMARK_MXR.

After problem_cache.json file is generated by this script,
use MIGRAPHX_PROBLEM_CACHE=problem_cache.json to consume it.

Usage: benchmark_mxr.py <mxr_dir>
"""
Comment thread tools/benchmark_mxr.py
Comment on lines +23 to +25
text = next(i.op().values()["text"] for i in p.get_main_module()
if i.name() == "@comment")
meta = dict(json.loads(text))
@ahsan-ca ahsan-ca added the skip bot checks Skips the Performance and Accuracy CI tests label Jun 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip bot checks Skips the Performance and Accuracy CI tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants