feat: swe bench scorer#342
Open
tianmu-li wants to merge 5 commits into
Open
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces support for SWE-bench accuracy evaluation by adding a new accuracy-only SWEBench dataset, a SWEBenchScorer that runs evaluations using mini-swe-agent in an isolated environment, and associated configuration templates, tests, and runbooks. Feedback on the changes focuses on improving the robustness of the SWEBenchScorer implementation, specifically by safely handling missing or null values when parsing configuration templates, benchmark configurations, and evaluation results, as well as gracefully handling cases where the Docker binary is missing from the system's PATH during preflight checks.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
tianmu-li
force-pushed
the
feat/swe_bench_scorer
branch
2 times, most recently
from
aa7f559 to
bb9b307
Compare
There was a problem hiding this comment.
Pull request overview
Adds first-class SWE-bench accuracy support to the benchmarking system by introducing an external (subprocess-driven) scorer and a predefined SWE-bench dataset, plus wiring in config/schema handling and examples to run the workflow.
Changes:
- Introduces
SWEBenchScorer(external evaluation viauv run --project ... mini-extra+swebenchharness) and integrates it into the scoring/benchmark lifecycle (preflight, phase skipping, reporting). - Adds predefined
swe_benchdataset with caching + anACCURACY_ONLYguard to prevent using it as a performance dataset. - Updates schema/templates/examples and adds unit tests for the scorer + dataset + benchmark wiring.
Reviewed changes
Copilot reviewed 20 out of 21 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| uv.lock | Locks pyyaml as a direct dependency. |
| pyproject.toml | Adds pyyaml==6.0.3 to runtime dependencies. |
| src/inference_endpoint/evaluation/scoring.py | Adds scorer preflight hooks, shared subprocess logging helper, and new SWEBenchScorer implementation. |
| src/inference_endpoint/dataset_manager/predefined/swe_bench/init.py | Adds SWE-bench predefined dataset with HF download + parquet cache and ACCURACY_ONLY=True. |
| src/inference_endpoint/dataset_manager/dataset.py | Introduces Dataset.ACCURACY_ONLY class flag. |
| src/inference_endpoint/dataset_manager/init.py | Exposes/imports SWEBench dataset. |
| src/inference_endpoint/config/schema.py | Adds swe_bench_scorer to ScorerMethod and injects concurrency → SWE-bench workers default. |
| src/inference_endpoint/commands/benchmark/execute.py | Runs scorer preflight(), prevents accuracy-only datasets as perf, skips endpoint phases for external scorers, and improves sample counting/reporting. |
| src/inference_endpoint/config/templates/online_template_full.yaml | Documents swe_bench_scorer as an accuracy scorer option. |
| src/inference_endpoint/config/templates/offline_template_full.yaml | Documents swe_bench_scorer as an accuracy scorer option. |
| src/inference_endpoint/config/templates/concurrency_template_full.yaml | Documents swe_bench_scorer as an accuracy scorer option. |
| examples/10_Agentic_Inference/swebench_template.yaml | Adds a SWE-bench/mini-swe-agent config template used by the external scorer. |
| examples/10_Agentic_Inference/swe_bench_accuracy.yaml | Adds a runnable example config for SWE-bench external accuracy evaluation. |
| examples/10_Agentic_Inference/README.md | Documents how to sync/run the SWE-bench accuracy subproject. |
| examples/10_Agentic_Inference/qwen_agentic_benchmark.yaml | Adds SWE-bench accuracy dataset to an agentic benchmark example. |
| examples/10_Agentic_Inference/kimi_agentic_benchmark.yaml | Adds SWE-bench accuracy dataset to an agentic benchmark example. |
| examples/10_Agentic_Inference/accuracy/RUNBOOK.md | Adds an ops runbook for validating the SWE-bench accuracy pipeline. |
| examples/10_Agentic_Inference/accuracy/pyproject.toml | Adds isolated uv subproject deps for SWE-bench evaluation tooling. |
| tests/unit/evaluation/test_swe_bench_scorer.py | Adds unit coverage for scorer behavior, config patching, and preflight. |
| tests/unit/dataset_manager/test_swe_bench_dataset.py | Adds unit coverage for dataset registration, caching, and subset mapping. |
| tests/unit/commands/test_benchmark.py | Adds tests for concurrency→workers injection, ACCURACY_ONLY enforcement, preflight propagation, and phase skipping behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tianmu-li
force-pushed
the
feat/swe_bench_scorer
branch
from
8d66f43 to
ae04494
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #342 +/- ##
=======================================
Coverage ? 80.87%
=======================================
Files ? 116
Lines ? 15969
Branches ? 0
=======================================
Hits ? 12915
Misses ? 3054
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
SWEBenchScorer.score() only evaluates min(num_instances, n_rows), but results.json was recording the full loaded dataset size. Also default external_sample_count to DEFAULT_NUM_INSTANCES instead of None when num_instances is omitted, since that's the common case.
Copilot review flagged three robustness gaps in SWEBenchScorer: score() read model_params.name via a raw subscript (KeyError on malformed config.yaml) instead of the guarded check _patch_config() already used, and _derive_required_images()/_prepull_image() let subprocess.TimeoutExpired bubble up raw instead of as a SetupError. Also adds targeted unit tests for previously-uncovered SWE-bench branches (preflight error paths, toolcall patch install/restore failure handling, _patch_config endpoint branches, module-level helpers) to close the patch-coverage gap codecov reported on scoring.py. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Type of change
Related issues
Testing
Checklist