feat(metrics): consolidate gpt-oss/deepseek accuracy, dedup reporting, surface TPS#400
Open
arekay-nv wants to merge 9 commits into
Open
feat(metrics): consolidate gpt-oss/deepseek accuracy, dedup reporting, surface TPS#400arekay-nv wants to merge 9 commits into
arekay-nv wants to merge 9 commits into
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces support for consolidating multiple accuracy datasets into a single combined score using a consolidation group tag, which is particularly useful for rolling up subsets like gpt-oss (e.g., AIME25, GPQA, and LiveCodeBench) into a sample-weighted overall score. It extracts shared accuracy breakdown helpers into a new module, refactors compliance and plotting utilities to use them, and updates the benchmark finalization process to attach consolidated accuracy breakdowns to the report. The feedback suggests a cleaner and safer approach to retrieving the dataset sample count during per-subset audit entry generation by avoiding fragile assert statements and direct internal attribute access in favor of the public num_samples() API.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
…, surface TPS Deduplicate the reporting layer, give gpt-oss-120b and deepseek-r1 a uniform per-subset accuracy breakdown, and surface TPS everywhere QPS appears. - Add shared evaluation/accuracy_results.py (to_float, find_accuracy_breakdown, ACCURACY_METRIC_KEYS, build_breakdown); metrics/results_plots.py and compliance/checker.py import it instead of copy-pasted helpers. - DeepSeek-R1 scorer surfaces its per-subset breakdown via score_breakdown() (BFCL-shaped: overall_accuracy/subset_scores/total_samples, 0-100). - Add GptOssAccuracyScorer: runs the 3 gpt-oss member scorers once and rolls them up sample-weighted by unique problem count. finalize emits both the per-subset <subset>::gptoss entries (audit + downstream compat) and one consolidated `gptoss` entry with a breakdown, from the single scoring pass. - Consolidation is opt-in via explicit accuracy_config.group; the ::variant suffix is a naming convention, not a grouping signal. Group members must return scalar scores in [0,1]. - Add accuracy to Report.display()/report.txt/result_summary.json; write the report in a finally so a scoring failure can't discard perf artifacts. - Single-source results.json qps/tps from Report (drop the duplicate QPS recompute); add tps + a symmetric "TPS: N/A" display branch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
arekay-nv
force-pushed
the
arekay/feat-consolidate-accuracy-reporting
branch
from
213b8c5 to
1ecb9e8
Compare
nvzhihanj
reviewed
Jul 8, 2026
arekay-nv
changed the title
nvzhihanj
reviewed
Jul 8, 2026
nvzhihanj
reviewed
Jul 8, 2026
…scorer Add a single-entry composite path for gpt-oss-120b accuracy, mirroring the DeepSeek-R1 model: one predefined dataset (`gptoss_120b_accuracy`) bundles the three subsets (aime25/gpqa/livecodebench) behind a `subset` routing column, and one registered scorer (`GptOss120bAccuracyScorer`, REQUIRES_EXTRACTOR=False) routes per-subset grading in-process (aime->BoxedMath/exact, gpqa->ABCD/exact, livecodebench->PythonCode via the lcb-service container), rolling the per-subset means into one headline sample-weighted by unique problem count plus a per-subset breakdown. A config picks EITHER this composite dataset (one results.json entry with a headline `score` on the 0-100 scale + `breakdown`) OR the existing individual `::gptoss` datasets under `group: gptoss` (separate entries). Both paths coexist unchanged. - scoring.py: `GptOss120bAccuracyScorer` + shared helpers `_exact_match` and `_weighted_breakdown` (dedup PassAt1Scorer and GptOssAccuracyScorer, both kept behaviour-identical). - dataset: `GptOss120bAccuracy` composes the 3 subsets at build time (each subset's own `gptoss()` preset renders `prompt`), caches a parquet; no committed artifact. num_samples stays the unique count; `max_samples` yields a stratified subset. Uniform num_repeats (per-subset 8/5/3 stays on the individual path). - schema: `ScorerMethod.GPTOSS_120B_ACCURACY` (+ regenerated templates). - example: examples/04_GPTOSS120B_Example/gptoss_120b_accuracy_example.yaml. - tests: composite scorer (routing/weighting/0-100 scale/lcb-down), composite dataset (schema/rendering/caching), and example-resolve coexistence. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ded by composite scorer)
…try per dataset Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
arekay-nv
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Deduplicates the reporting layer, gives gpt-oss-120b and deepseek-r1 a uniform per-subset accuracy breakdown, and surfaces TPS everywhere QPS appears.
Reporting dedup
evaluation/accuracy_results.py(to_float,find_accuracy_breakdown,ACCURACY_METRIC_KEYS,build_breakdown);metrics/results_plots.pyandcompliance/checker.pynow import it instead of their copy-pasted helpers.results.jsonresultsblock is single-sourced from theReport(drops the duplicate QPS recompute inexecute.py).Accuracy consolidation (uniform BFCL-shaped breakdown:
overall_accuracy/subset_scores/total_samples, 0–100)LegacyMLPerfDeepSeekR1Scorernow surfaces its per-subset breakdown viascore_breakdown()(was discarded to a scratch file).GptOssAccuracyScorerruns the 3 member scorers once and rolls them up sample-weighted by unique problem count (matchingendpoints-launch/…/generate_points.py).finalize_benchmarkemits both the per-subset<subset>::gptossentries (audit + downstream compat) and one consolidatedgptossentry with a breakdown — from the single scoring pass, no re-scoring.accuracy_config.group(the::variantsuffix is a naming convention, not a grouping signal). Group members must return scalar scores in[0,1].Report.display()/report.txt/result_summary.json; the report is written in afinallyso a scoring failure can't discard the perf run's artifacts.TPS
results.json;Report.display()gains a symmetricTPS: N/Abranch.Test plan
test_accuracy_results.py,test_gptoss_scorer.py,test_score_accuracy.py; extended deepseek/report/plots/checker tests. 412 unit tests pass; all pre-commit hooks pass.🤖 Generated with Claude Code