feat: swe bench scorer

AGENTS.md

@@ -99,6 +99,7 @@ Dataset Manager --> Load Generator --> Endpoint Client --> External Endpoint
 | **TensorRT-LLM**                    | `src/inference_endpoint/trtllm/`                                                                                   | Adapter for TensorRT-LLM endpoints. `TRTLLMAdapter` sends requests; `TRTLLMSSEAccumulator` handles SSE streaming responses.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
 | **DeepSeek-R1 (MLPerf)**            | `src/inference_endpoint/evaluation/scoring.py` (`LegacyMLPerfDeepSeekR1Scorer`), `examples/07_DeepSeekR1_Example/` | MLPerf DeepSeek-R1 accuracy. TensorRT-LLM is OpenAI-compatible, so it is served via `api_type: openai` / `openai_completions` (no dedicated trtllm adapter). The combined multi-subset eval (`math500`/`aime`/`gpqa`/`mmlu_pro`/`livecodebench`) is the official MLCommons `eval_accuracy.py`, run out-of-process via `uv run --project` against the isolated subproject at `src/inference_endpoint/evaluation/legacy_mlperf_deepseek_r1/` (a uv subproject excluded from the parent wheel; mirrors the VBench pattern). The example feeds the exact MLPerf prompt via pre-tokenized `input_tokens` to `/v1/completions`.                                                                                                                                                      |
 | **VideoGen**                        | `src/inference_endpoint/videogen/`                                                                                 | Adapter for video-generation endpoints (e.g. trtllm-serve `POST /v1/videos/generations`, used by MLPerf WAN2.2-T2V-A14B). Defaults to `response_format=video_path` (server saves video to shared storage and returns path) to avoid large byte payloads. Accuracy mode also runs on `video_path`: the adapter mirrors the path into `response_output` so the event log carries it to `VBenchScorer` (see `evaluation/scoring.py`), which scores videos via VBench from a sibling `uv` subproject at `examples/09_Wan22_VideoGen_Example/accuracy/` (vbench's `transformers==4.33.2` + `numpy<2` pins are incompatible with the parent env, so it runs out-of-process via `uv run --project`). Dataset is ingested via the generic JSONL loader.                                |
+| **SWE-bench**                       | `src/inference_endpoint/dataset_manager/predefined/swe_bench/`, `evaluation/scoring.py`                            | `SWEBench` predefined dataset (HuggingFace `princeton-nlp/SWE-bench_Verified` or `_Lite`; `ACCURACY_ONLY=True`). `SWEBenchScorer` sets `SKIP_ENDPOINT_PHASE=True` and bypasses the built-in accuracy phase entirely: it calls `mini-extra swebench` then `swebench.harness.run_evaluation` via `uv run --project examples/10_Agentic_Inference/accuracy/`. Qwen configs may opt into `enable_swebench_toolcall_patch: true`, which builds a temporary minisweagent package overlay with the shipped `finish` and `str_replace_editor` replacements, prepends it to `PYTHONPATH` for the agent run, and leaves the installed package untouched. Kimi and default runs leave this flag unset.                                                                                    |
 | **Compliance (submission checker)** | `src/inference_endpoint/compliance/checker.py`, `scripts/check_compliance.py`                                      | Validates a completed run's report directory against a registered ruleset. `check_submission(report_dir, ruleset, model)` reads the resolved `config.yaml` plus scorer output (`results.json` for accuracy, `scores.json` for the agentic perf run) and runs config-lock (deterministic + single-stream), the accuracy gate (`score >= factor x reference`, factor 0.97 for Edge-Agentic), and run validity (0 dropped turns). Server-side launch flags (`--reasoning off`, `--ctx-size`) aren't in client artifacts, so they're surfaced as manual attestations. CLI: `scripts/check_compliance.py REPORT_DIR` (exit 0 = pass).                                                                                                                                               |
 | **Compliance (audit tests)**        | `src/inference_endpoint/compliance/`, `commands/audit.py`                                                          | MLPerf compliance audits. `AuditTest` protocol + `AuditRunSpec`/`AuditRunArtifacts` + registry (`compliance/__init__.py`); `OutputCachingAudit` (`compliance/audit_test/output_caching_test.py`, which also owns the QPS-specific `AuditRunStats`) implements MLPerf **TEST04** output-caching detection — reference phase (distinct samples) vs. fixed-sample audit phase, comparing QPS against `threshold`. `commands/audit.py:run_audit` runs phases via `AuditTest.plan_runs`/`validate`, writing `audit_result.json`/`verify_<TEST>.txt` atomically via `compliance/result.py`. Enabled by the `audit:` YAML block; `cli._run` runs it after the main benchmark (upstream MLPerf order: perf run, then TEST04), or standalone with `audit.only: true`. Performance-only. |
 

examples/10_Agentic_Inference/README.md

@@ -194,3 +194,42 @@ Update the first `datasets` entry (`name` and `path`), `model_params.name`, and
 uv run inference-endpoint benchmark from-config \
   --config examples/10_Agentic_Inference/kimi_agentic_benchmark.yaml
 ```
+
+## SWE-bench Accuracy
+
+`swe_bench_accuracy.yaml` runs the SWE-bench accuracy evaluation alongside a
+minimal performance dataset. The benchmark framework skips its built-in
+accuracy phase for this dataset; instead, `SWEBenchScorer` shells out to
+`mini-swe-agent` and the `swebench` evaluation harness, and that external flow
+drives requests to the configured endpoint.
+
+Keep `accuracy_config.num_repeats: 1`: the scorer performs one external
+evaluation run per benchmark. Optional `accuracy_config.extras.subset` and
+`split` are used consistently for dataset loading, preflight, and scoring.
+
+`accuracy_config.extras.workers` sets the agent run's parallelism (`--workers`).
+If unset, it defaults to the load pattern's `target_concurrency` (for
+`concurrency`/`agentic_inference` patterns), else 10. `max_eval_workers`
+(default 10, `--max_workers`) sets the eval harness's parallelism.
+
+The isolated `uv` environment for those tools lives in `accuracy/`. Sync it
+once before running:
+
+```bash
+cd examples/10_Agentic_Inference/accuracy
+uv sync
+```
+
+Then run the benchmark from the repo root:
+
+```bash
+uv run inference-endpoint benchmark from-config \
+  --config examples/10_Agentic_Inference/swe_bench_accuracy.yaml \
+  --mode both
+```
+
+`--mode both` is required: `type: online` configs default to `TestMode.PERF`,
+which skips accuracy datasets.
+
+See `accuracy/RUNBOOK.md` for preconditions, sanity checks, and common failure
+modes.

examples/10_Agentic_Inference/accuracy/RUNBOOK.md

@@ -0,0 +1,66 @@
+# SWE-bench Accuracy Smoke-Test Runbook
+
+End-to-end validation for the SWE-bench accuracy pipeline. Unit tests mock all
+subprocesses, so running the real pipeline is the only way to catch Docker,
+HuggingFace access, or mini-swe-agent wiring issues.
+
+## 0. Preconditions
+
+- Docker daemon running (swebench harness spawns one container per instance).
+- Docker Hub auth or a pre-seeded image cache for uncached SWE-bench images.
+- Network egress to PyPI and HuggingFace Hub.
+- `uv` binary on PATH (`curl -LsSf https://astral.sh/uv/install.sh | sh`).
+- Parent endpoints env already synced (`uv sync --extra dev` from repo root).
+
+## 1. Sync the accuracy subproject
+
+From the repo root:
+
+```bash
+cd examples/10_Agentic_Inference/accuracy
+uv sync
+```
+
+Sanity check:
+
+```bash
+uv run mini-extra --help
+uv run python -m swebench.harness.run_evaluation --help
+```
+
+Override the default subproject path via env var if needed:
+
+```bash
+export SWE_BENCH_PROJECT_PATH="$(pwd)/examples/10_Agentic_Inference/accuracy"
+```
+
+## 2. End-to-end test (requires live endpoint)
+
+```bash
+uv run inference-endpoint benchmark from-config \
+  --config examples/10_Agentic_Inference/swe_bench_accuracy.yaml \
+  --mode both
+```
+
+`--mode both` is required: `type: online` configs default to `TestMode.PERF`,
+which skips accuracy datasets.
+
+Scorer preflight resolves the requested SWE-bench instances and pre-pulls the
+required Docker images before `mini-extra swebench` starts, using the configured
+SWE-bench `workers` count and a compact full-count progress bar. Cached images
+still complete immediately in that bar.
+
+Qwen SWE-bench configs may opt into `enable_swebench_toolcall_patch: true` and
+the `swebench_qwen_tools_template.yaml` template. That path builds a temporary
+minisweagent package overlay with the replacement files shipped in this
+subproject, prepends it to `PYTHONPATH` for the agent run, and leaves the
+installed package untouched. Leave this flag unset for Kimi and other non-Qwen
+runs.
+
+## Common failure modes
+
+| Symptom                                              | Likely cause                          | Fix                                                       |
+| ---------------------------------------------------- | ------------------------------------- | --------------------------------------------------------- |
+| `FileNotFoundError: SWE-bench subproject not found`  | subproject not synced                 | Run `uv sync` in `examples/10_Agentic_Inference/accuracy` |
+| Docker error during `run_evaluation`                 | Docker daemon not running             | Start Docker and retry                                    |
+| `Failed to pre-pull required SWE-bench Docker image` | Docker Hub rate limit or missing auth | Run `docker login` or use a local image cache/mirror      |