diff --git a/AGENTS.md b/AGENTS.md index 9c133348f..d8b69d5c4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -100,6 +100,7 @@ Dataset Manager --> Load Generator --> Endpoint Client --> External Endpoint | **DeepSeek-R1 (MLPerf)** | `src/inference_endpoint/evaluation/scoring.py` (`LegacyMLPerfDeepSeekR1Scorer`), `examples/07_DeepSeekR1_Example/` | MLPerf DeepSeek-R1 accuracy. TensorRT-LLM is OpenAI-compatible, so it is served via `api_type: openai` / `openai_completions` (no dedicated trtllm adapter). The combined multi-subset eval (`math500`/`aime`/`gpqa`/`mmlu_pro`/`livecodebench`) is the official MLCommons `eval_accuracy.py`, run out-of-process via `uv run --project` against the isolated subproject at `src/inference_endpoint/evaluation/legacy_mlperf_deepseek_r1/` (a uv subproject excluded from the parent wheel; mirrors the VBench pattern). The example feeds the exact MLPerf prompt via pre-tokenized `input_tokens` to `/v1/completions`. | | **VideoGen** | `src/inference_endpoint/videogen/` | Adapter for video-generation endpoints (e.g. trtllm-serve `POST /v1/videos/generations`, used by MLPerf WAN2.2-T2V-A14B). Defaults to `response_format=video_path` (server saves video to shared storage and returns path) to avoid large byte payloads. Accuracy mode also runs on `video_path`: the adapter mirrors the path into `response_output` so the event log carries it to `VBenchScorer` (see `evaluation/scoring.py`), which scores videos via VBench from a sibling `uv` subproject at `examples/09_Wan22_VideoGen_Example/accuracy/` (vbench's `transformers==4.33.2` + `numpy<2` pins are incompatible with the parent env, so it runs out-of-process via `uv run --project`). Dataset is ingested via the generic JSONL loader. | | **Compliance (submission checker)** | `src/inference_endpoint/compliance/checker.py`, `scripts/check_compliance.py` | Validates a completed run's report directory against a registered ruleset. `check_submission(report_dir, ruleset, model)` reads the resolved `config.yaml` plus scorer output (`accuracy/accuracy_results.json` for accuracy, `scores.json` for the agentic perf run) and runs config-lock (deterministic + single-stream), the accuracy gate (`score >= factor x reference`, factor 0.97 for Edge-Agentic), and run validity (0 dropped turns). Server-side launch flags (`--reasoning off`, `--ctx-size`) aren't in client artifacts, so they're surfaced as manual attestations. CLI: `scripts/check_compliance.py REPORT_DIR` (exit 0 = pass). | +| **SWE-bench** | `src/inference_endpoint/dataset_manager/predefined/swe_bench/`, `evaluation/scoring.py` | `SWEBench` predefined dataset (HuggingFace `princeton-nlp/SWE-bench_Verified` or `_Lite`; `ACCURACY_ONLY=True`). `SWEBenchScorer` sets `SKIP_ENDPOINT_PHASE=True` and bypasses the built-in accuracy phase entirely: it delegates agent execution and grading to the configured SWE-bench service via `accuracy_config.extras.swebench_service_url`. The service host owns Docker/runtime execution, artifact storage, and secret handling; the benchmark client remains the report-producing entrypoint. | | **Compliance (audit tests)** | `src/inference_endpoint/compliance/`, `commands/audit.py` | MLPerf compliance audits. `AuditTest` protocol + `AuditRunSpec`/`AuditRunArtifacts` + registry (`compliance/__init__.py`); `OutputCachingAudit` (`compliance/audit_test/output_caching_test.py`, which also owns the QPS-specific `AuditRunStats`) implements MLPerf **TEST04** output-caching detection — reference phase (distinct samples) vs. fixed-sample audit phase, comparing QPS against `threshold`. `commands/audit.py:run_audit` runs phases via `AuditTest.plan_runs`/`validate`, writing `audit_result.json`/`verify_.txt` atomically via `compliance/result.py`. Enabled by the `audit:` YAML block; `cli._run` runs it after the main benchmark (upstream MLPerf order: perf run, then TEST04), or standalone with `audit.only: true`. Performance-only. | ### Hot-Path Architecture diff --git a/examples/10_Agentic_Inference/README.md b/examples/10_Agentic_Inference/README.md index ab3673b51..66094bdf2 100644 --- a/examples/10_Agentic_Inference/README.md +++ b/examples/10_Agentic_Inference/README.md @@ -194,3 +194,56 @@ Update the first `datasets` entry (`name` and `path`), `model_params.name`, and uv run inference-endpoint benchmark from-config \ --config examples/10_Agentic_Inference/kimi_agentic_benchmark.yaml ``` + +## SWE-bench Accuracy + +`swe_bench_accuracy.yaml` runs the SWE-bench accuracy evaluation alongside a +minimal performance dataset. The benchmark framework skips its built-in +accuracy phase for this dataset; instead, `SWEBenchScorer` submits the run to a +native SWE-bench service. The service host owns Docker, `mini-swe-agent`, and +the `swebench` evaluation harness, and it drives requests to the configured +endpoint. + +Keep `accuracy_config.num_repeats: 1`: the scorer performs one external +evaluation run per benchmark. Optional `accuracy_config.extras.subset` and +`split` are used consistently for dataset loading, preflight, and scoring. + +`accuracy_config.extras.swebench_service_url` points the benchmark client to +the service. Service mode follows the LiveCodeBench-style external-service +convention for heavyweight evaluation work and supports exactly one endpoint URL +in `endpoint_config.endpoints`; that URL must be reachable from the service +host. Treat the service host as trusted infrastructure: it receives the endpoint +URL and optional endpoint API key needed to run mini-swe-agent. +For non-loopback service deployments, bind it on a private network or start it +with `--auth-token` and set +`accuracy_config.extras.swebench_service_auth_token`. + +`accuracy_config.extras.workers` sets the agent run's parallelism (`--workers`). +If unset, it defaults to the load pattern's `target_concurrency` (for +`concurrency`/`agentic_inference` patterns), else 10. `max_eval_workers` +(default 10, `--max_workers`) sets the eval harness's parallelism. + +Qwen tool-call runs should set `enable_swebench_toolcall_patch: true` and +`swebench_template: qwen_tools`. Template contents are owned by the service; the +client does not send host file paths for templates or patch code. + +Start the service on the host that has Docker: + +```bash +uv run --project src/inference_endpoint/evaluation/swebench_service \ + python -m swebench_service --host 0.0.0.0 --port 18080 +``` + +Then run the benchmark from the repo root: + +```bash +uv run inference-endpoint benchmark from-config \ + --config examples/10_Agentic_Inference/swe_bench_accuracy.yaml \ + --mode both +``` + +`--mode both` is required: `type: online` configs default to `TestMode.PERF`, +which skips accuracy datasets. + +See `accuracy/RUNBOOK.md` for preconditions, sanity checks, and common failure +modes. diff --git a/examples/10_Agentic_Inference/accuracy/RUNBOOK.md b/examples/10_Agentic_Inference/accuracy/RUNBOOK.md new file mode 100644 index 000000000..52954aca6 --- /dev/null +++ b/examples/10_Agentic_Inference/accuracy/RUNBOOK.md @@ -0,0 +1,63 @@ +# SWE-bench Accuracy Smoke-Test Runbook + +End-to-end validation for the SWE-bench accuracy pipeline. Unit tests mock all +subprocesses, so running the real pipeline is the only way to catch Docker, +HuggingFace access, or mini-swe-agent wiring issues. + +## 0. Preconditions + +- Docker daemon running on the SWE-bench service host. +- Docker Hub auth or a pre-seeded image cache on the service host. +- Network egress to PyPI and HuggingFace Hub from the service host. +- Endpoint URL reachable from the service host. +- `uv` binary on PATH (`curl -LsSf https://astral.sh/uv/install.sh | sh`). +- Parent endpoints env already synced (`uv sync --extra dev` from repo root). + +## 1. Start the SWE-bench service + +From the repo root: + +```bash +uv run --project src/inference_endpoint/evaluation/swebench_service \ + python -m swebench_service --host 0.0.0.0 --port 18080 +``` + +Sanity check: + +```bash +curl http://localhost:18080/health +``` + +## 2. End-to-end test (requires live endpoint) + +```bash +uv run inference-endpoint benchmark from-config \ + --config examples/10_Agentic_Inference/swe_bench_accuracy.yaml \ + --mode both +``` + +`--mode both` is required: `type: online` configs default to `TestMode.PERF`, +which skips accuracy datasets. + +Scorer preflight calls the service `/health` endpoint. It does not check Docker +or pre-pull images on the benchmark client. + +The service is trusted infrastructure. It receives one endpoint URL and optional +endpoint credentials, runs Docker-backed evaluations, and serves artifacts. For +non-loopback deployments, bind it on a private network or start it with +`--auth-token TOKEN` and set +`accuracy_config.extras.swebench_service_auth_token: TOKEN`. + +Qwen SWE-bench configs may opt into `enable_swebench_toolcall_patch: true` and +`swebench_template: qwen_tools`. That path builds a temporary minisweagent +package overlay with replacement files packaged with the service, prepends it to +`PYTHONPATH` for the agent run, and leaves the installed package untouched. +Leave this flag unset for Kimi and other non-Qwen runs. + +## Common failure modes + +| Symptom | Likely cause | Fix | +| ------------------------------------ | ----------------------------------------- | -------------------------------------------------- | +| `swebench_service_url is required` | Client config missing service URL | Set `accuracy_config.extras.swebench_service_url` | +| Service health check fails | Service not running or unreachable | Start the service or fix client-to-service routing | +| Docker error during `run_evaluation` | Docker daemon not running on service host | Start Docker on the service host and retry | diff --git a/examples/10_Agentic_Inference/kimi_agentic_benchmark.yaml b/examples/10_Agentic_Inference/kimi_agentic_benchmark.yaml index 9740aa4c1..696b399bd 100644 --- a/examples/10_Agentic_Inference/kimi_agentic_benchmark.yaml +++ b/examples/10_Agentic_Inference/kimi_agentic_benchmark.yaml @@ -23,6 +23,15 @@ datasets: num_trajectories_to_issue: 990 # Should be integer multiple of 990. # Required benchmark default; set to true only for faster optimization/debug runs. stop_issuing_on_first_user_complete: false + - name: swe_bench + type: "accuracy" + accuracy_config: + eval_method: "swe_bench_scorer" + num_repeats: 1 + extras: + swebench_service_url: "http://localhost:18080" + num_instances: 200 + # workers: 8 # parallel agent workers; defaults to target_concurrency if unset. settings: runtime: diff --git a/examples/10_Agentic_Inference/qwen_agentic_benchmark.yaml b/examples/10_Agentic_Inference/qwen_agentic_benchmark.yaml new file mode 100644 index 000000000..6542ef3f0 --- /dev/null +++ b/examples/10_Agentic_Inference/qwen_agentic_benchmark.yaml @@ -0,0 +1,53 @@ +name: "qwen-agentic-benchmark" +version: "1.0" +type: "online" + +model_params: + name: "Qwen/Qwen3.6-35B-A3B" + temperature: 1.0 + top_k: 20 + top_p: 0.95 + repetition_penalty: 1.0 + presence_penalty: 1.5 + max_new_tokens: 8192 + chat_template_kwargs: + preserve_thinking: true + +datasets: + - name: agentic_coding + type: performance + path: ./agentic_combined.jsonl + accuracy_config: + eval_method: agentic_inference_inline # required benchmark default. + agentic_inference: + turn_timeout_s: 14400.0 + enable_salt: true # do not change. + inject_tool_delay: true # do not change. + - name: swe_bench + type: "accuracy" + accuracy_config: + eval_method: "swe_bench_scorer" + num_repeats: 1 + extras: + swebench_service_url: "http://localhost:18080" + num_instances: 200 + workers: 10 + max_eval_workers: 10 + enable_swebench_toolcall_patch: true + swebench_template: "qwen_tools" + +settings: + runtime: + min_duration_ms: 0 + max_duration_ms: 36000000 + + load_pattern: + type: agentic_inference + target_concurrency: 8 # Submission-specific concurrency. + +endpoint_config: + endpoints: + - "http://localhost:30000" + api_type: openai + +report_dir: logs/qwen_agentic diff --git a/examples/10_Agentic_Inference/swe_bench_accuracy.yaml b/examples/10_Agentic_Inference/swe_bench_accuracy.yaml new file mode 100644 index 000000000..3f0dcc315 --- /dev/null +++ b/examples/10_Agentic_Inference/swe_bench_accuracy.yaml @@ -0,0 +1,47 @@ +type: "online" + +model_params: + name: "Qwen/Qwen3.6-35B-A3B" + temperature: 1.0 + top_p: 0.95 + top_k: 20 + repetition_penalty: 1.0 + presence_penalty: 1.5 + max_new_tokens: 8192 + chat_template_kwargs: + preserve_thinking: true + +datasets: + # Minimal performance dataset required by the framework. + - name: swe_bench_perf + type: "performance" + path: "tests/assets/datasets/dummy_1k.jsonl" + parser: + prompt: text_input + + # Accuracy dataset — instance_id rows tell mini-swe-agent which instances to run. + # First run downloads ~10 MB from HuggingFace and caches to datasets_dir. + - name: swe_bench + type: "accuracy" + accuracy_config: + eval_method: "swe_bench_scorer" + num_repeats: 1 # SWE-bench runs one external evaluation pass per benchmark. + extras: + swebench_service_url: "http://localhost:18080" + num_instances: 200 + enable_swebench_toolcall_patch: true + swebench_template: "qwen_tools" + # subset: "lite" + # split: "test" + +settings: + load_pattern: + type: "concurrency" + target_concurrency: 10 # auto-forwarded as workers= to swe_bench_scorer by the config validator + runtime: + n_samples_to_issue: 10 + +endpoint_config: + endpoints: + - "http://localhost:30000" + api_type: "openai" diff --git a/pyproject.toml b/pyproject.toml index b223dfd37..3a35afc03 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -33,6 +33,8 @@ source-exclude = [ # numpy<2 / prm800k that conflict with the parent env). Invoked only via # `uv run --project`, never imported - keep it out of the parent wheel. "inference_endpoint/evaluation/legacy_mlperf_deepseek_r1/**", + # Isolated native uv service that owns SWE-bench Docker execution. + "inference_endpoint/evaluation/swebench_service/**", ] [project] @@ -94,6 +96,7 @@ dependencies = [ # Fix pytz-2024 import warning "pytz==2026.1.post1", "urllib3==2.7.0", + "pyyaml==6.0.3", ] [project.optional-dependencies] diff --git a/src/inference_endpoint/commands/benchmark/execute.py b/src/inference_endpoint/commands/benchmark/execute.py index 301c1781e..2852ebcb5 100644 --- a/src/inference_endpoint/commands/benchmark/execute.py +++ b/src/inference_endpoint/commands/benchmark/execute.py @@ -76,6 +76,7 @@ LoadPattern, LoadPatternType, ProfilerEngine, + ScorerMethod, StreamingMode, TestMode, TestType, @@ -186,6 +187,15 @@ class AccuracyConfiguration: dataset_type: DatasetType = DatasetType.ACCURACY +def _effective_external_sample_count( + eval_cfg: AccuracyConfiguration, +) -> int | None: + count = eval_cfg.scorer.external_sample_count(eval_cfg.extras) + if count is None: + return None + return min(count, eval_cfg.dataset.num_samples()) + + @dataclass class BenchmarkContext: """All state needed to run a benchmark, created by setup_benchmark. @@ -298,6 +308,34 @@ def _resolve_accuracy_components( return scorer_cls, extractor_cls +def _validate_accuracy_config_for_scorer( + scorer_cls: type[Scorer], + dataset_name: str, + accuracy_config: Any, +) -> None: + if ( + scorer_cls.SCORER_ID == ScorerMethod.SWE_BENCH.value + and accuracy_config.num_repeats != 1 + ): + raise InputValidationError( + f"Dataset '{dataset_name}' uses scorer '{scorer_cls.SCORER_ID}'; " + "accuracy_config.num_repeats must be 1 because SWE-bench evaluation " + "runs externally once per benchmark." + ) + + +def _run_scorer_preflight( + scorer_cls: type[Scorer], + extras: dict[str, Any], + *, + loaded_sample_count: int | None = None, +) -> None: + if scorer_cls.SCORER_ID == ScorerMethod.SWE_BENCH.value: + scorer_cls.preflight(extras, loaded_sample_count=loaded_sample_count) + else: + scorer_cls.preflight(extras) + + def _load_datasets( config: BenchmarkConfig, report_dir: Path, @@ -320,17 +358,30 @@ def _load_datasets( accuracy_datasets: list[Dataset] = [] eval_configs: list[AccuracyConfiguration] = [] + load_accuracy = test_mode in (TestMode.ACC, TestMode.BOTH) # Pack the evaluation parameters for each accuracy dataset - for acc_cfg in accuracy_cfgs: + accuracy_cfgs_to_load = accuracy_cfgs if load_accuracy else [] + for acc_cfg in accuracy_cfgs_to_load: scorer_cls, extractor_cls = _resolve_accuracy_components( acc_cfg.name, acc_cfg.accuracy_config ) assert acc_cfg.accuracy_config is not None + _validate_accuracy_config_for_scorer( + scorer_cls, acc_cfg.name, acc_cfg.accuracy_config + ) + extras = acc_cfg.accuracy_config.extras or {} + ds = DataLoaderFactory.create_loader( - acc_cfg, num_repeats=acc_cfg.accuracy_config.num_repeats + acc_cfg, + num_repeats=acc_cfg.accuracy_config.num_repeats, + **scorer_cls.dataset_loader_kwargs(extras), ) + ds_model_params = acc_cfg.effective_generation_config(config.model_params) + ds.load(api_type=config.endpoint_config.api_type, model_params=ds_model_params) + logger.info(f"Loaded {ds} - {ds.num_samples()} samples") + _run_scorer_preflight(scorer_cls, extras, loaded_sample_count=ds.num_samples()) accuracy_datasets.append(ds) # TODO add tests and defaults eval_configs.append( @@ -346,14 +397,8 @@ def _load_datasets( dataset_type=DatasetType.ACCURACY, ) ) - # Value/api-type validity of the override is already enforced at config - # construction (BenchmarkConfig validates each dataset's effective params), - # so this cannot raise for a validated config. - ds_model_params = acc_cfg.effective_generation_config(config.model_params) - ds.load(api_type=config.endpoint_config.api_type, model_params=ds_model_params) - logger.info(f"Loaded {ds} - {ds.num_samples()} samples") - if not accuracy_cfgs: + if not accuracy_cfgs and load_accuracy: logger.info("No separate accuracy datasets provided") dataloader: Dataset | None = None @@ -364,6 +409,14 @@ def _load_datasets( if len(performance_cfgs) > 1: raise InputValidationError("Multiple performance datasets not supported") perf_cfg = performance_cfgs[0] + perf_base_name = perf_cfg.name.split("::")[0] + perf_cls = Dataset.PREDEFINED.get(perf_base_name) + if perf_cls is not None and perf_cls.ACCURACY_ONLY: + raise InputValidationError( + f"Dataset '{perf_cfg.name}' is accuracy-only and cannot be used " + "as a performance dataset. Use a different dataset (e.g. 'random') " + "for the performance phase." + ) # Override validity is enforced at config construction (see accuracy loop). perf_model_params = perf_cfg.effective_generation_config(config.model_params) try: @@ -380,7 +433,7 @@ def _load_datasets( except Exception as e: raise SetupError(f"Failed to load dataset: {e}") from e - if perf_cfg.accuracy_config is not None: + if load_accuracy and perf_cfg.accuracy_config is not None: accuracy_config = perf_cfg.accuracy_config if accuracy_config.num_repeats != 1: raise InputValidationError( @@ -391,6 +444,14 @@ def _load_datasets( scorer_cls, extractor_cls = _resolve_accuracy_components( perf_cfg.name, accuracy_config ) + _validate_accuracy_config_for_scorer( + scorer_cls, perf_cfg.name, accuracy_config + ) + _run_scorer_preflight( + scorer_cls, + accuracy_config.extras or {}, + loaded_sample_count=dataloader.num_samples(), + ) eval_configs.append( AccuracyConfiguration( @@ -497,8 +558,11 @@ def setup_benchmark( ) total_samples = rt_settings.total_samples_to_issue() - if accuracy_datasets: - total_samples += sum(ds.num_samples() * ds.repeats for ds in accuracy_datasets) + total_samples += sum( + ec.dataset.num_samples() * ec.dataset.repeats + for ec in eval_configs + if not ec.scorer.SKIP_ENDPOINT_PHASE and ec.dataset_name != "performance" + ) collect_responses = test_mode in (TestMode.ACC, TestMode.BOTH) logger.info( @@ -510,6 +574,16 @@ def setup_benchmark( ) else: logger.info(f"Accuracy-only mode, Expected samples: {total_samples}") + for ec in eval_configs: + if ec.scorer.SKIP_ENDPOINT_PHASE: + n = _effective_external_sample_count(ec) + if n is not None: + logger.info( + "Accuracy dataset '%s' (%s): %d instances evaluated externally", + ec.dataset_name, + ec.scorer.SCORER_ID, + n, + ) return BenchmarkContext( config=config, @@ -589,6 +663,8 @@ def _build_phases( # Accuracy phases — use eval_cfg.dataset_name as phase name so it matches # what Scorer._load_sample_index_map() looks up in sample_idx_map.json for eval_cfg in ctx.eval_configs: + if eval_cfg.scorer.SKIP_ENDPOINT_PHASE: + continue if eval_cfg.dataset_type == DatasetType.PERFORMANCE: continue acc_ds = eval_cfg.dataset @@ -1217,6 +1293,9 @@ def _write_scoring_artifacts( sample_idx_map: dict[str, dict[str, int]] = {} for phase_result in result.phase_results: sample_idx_map[phase_result.name] = phase_result.uuid_to_index + for eval_cfg in ctx.eval_configs: + if eval_cfg.scorer.SKIP_ENDPOINT_PHASE: + sample_idx_map.setdefault(eval_cfg.dataset_name, {}) map_path = ctx.report_dir / "sample_idx_map.json" with map_path.open("wb") as f: @@ -1359,6 +1438,8 @@ def _score_accuracy( issued counts instead of unit × repeats (repeats is forced to 1 there). """ accuracy_scores: list[dict[str, Any]] = [] + if ctx.test_mode not in (TestMode.ACC, TestMode.BOTH): + return accuracy_scores # Per-phase wall-clock (seconds) keyed by phase name. The accuracy phase name # is the dataset_name; the inline-scored perf entry keys on "performance". @@ -1443,6 +1524,11 @@ def _score_accuracy( total_samples = sum(phase.issued_count for phase in result.perf_results) else: total_samples = unit_samples * num_repeats + if eval_cfg.scorer.SKIP_ENDPOINT_PHASE: + ext = _effective_external_sample_count(eval_cfg) + if ext is not None: + unit_samples = ext + total_samples = ext entry: dict[str, Any] = { "dataset_name": eval_cfg.dataset_name, "extractor": ( diff --git a/src/inference_endpoint/config/schema.py b/src/inference_endpoint/config/schema.py index c60498540..fd0e5c1e1 100644 --- a/src/inference_endpoint/config/schema.py +++ b/src/inference_endpoint/config/schema.py @@ -147,6 +147,7 @@ class ScorerMethod(str, Enum): VBENCH = "vbench" BFCL_V4 = "bfcl_v4" LEGACY_MLPERF_DEEPSEEK_R1 = "legacy_mlperf_deepseek_r1" + SWE_BENCH = "swe_bench_scorer" class AuditTestId(str, Enum): @@ -1235,6 +1236,34 @@ def _resolve_and_validate(self) -> Self: f"got '{lp.type}'" ) + # For swe_bench_scorer, forward target_concurrency as workers when the + # user has not set it explicitly. mini-swe-agent's parallelism should + # match the endpoint's concurrency budget. + concurrency = ( + lp.target_concurrency + if lp.type + in (LoadPatternType.CONCURRENCY, LoadPatternType.AGENTIC_INFERENCE) + and lp.target_concurrency + else None + ) + if concurrency is not None and self.datasets: + updated_datasets = [] + changed = False + for ds in self.datasets: + acc = ds.accuracy_config + if ( + acc is not None + and acc.eval_method == ScorerMethod.SWE_BENCH + and (acc.extras is None or acc.extras.get("workers") is None) + ): + new_extras = {**(acc.extras or {}), "workers": concurrency} + new_acc = acc.model_copy(update={"extras": new_extras}) + ds = ds.model_copy(update={"accuracy_config": new_acc}) + changed = True + updated_datasets.append(ds) + if changed: + object.__setattr__(self, "datasets", updated_datasets) + # Pin RNG seeds from the submission ruleset. Done last so the values # are baked into the config before any consumer reads them — the config # dump to the report dir, RuntimeSettings.from_config, and the report diff --git a/src/inference_endpoint/config/templates/concurrency_template_full.yaml b/src/inference_endpoint/config/templates/concurrency_template_full.yaml index 521f6f335..e16700d75 100644 --- a/src/inference_endpoint/config/templates/concurrency_template_full.yaml +++ b/src/inference_endpoint/config/templates/concurrency_template_full.yaml @@ -43,7 +43,7 @@ datasets: # Dataset configs system: system_prompt generate_params: null # Dataset-specific parameters passed to the generate() method accuracy_config: # Accuracy evaluation settings - eval_method: pass_at_1 # Scorer method | options: pass_at_1, string_match, rouge, code_bench_scorer, shopify_category_f1, agentic_inference_inline, vbench, bfcl_v4, legacy_mlperf_deepseek_r1 + eval_method: pass_at_1 # Scorer method | options: pass_at_1, string_match, rouge, code_bench_scorer, shopify_category_f1, agentic_inference_inline, vbench, bfcl_v4, legacy_mlperf_deepseek_r1, swe_bench_scorer ground_truth: ground_truth # Ground truth column name extractor: boxed_math_extractor # Answer extractor (abcd_extractor, boxed_math_extractor, identity_extractor, python_code_extractor) num_repeats: 1 # Repeat dataset N times for evaluation diff --git a/src/inference_endpoint/config/templates/offline_template_full.yaml b/src/inference_endpoint/config/templates/offline_template_full.yaml index 51e06c738..a18c5d091 100644 --- a/src/inference_endpoint/config/templates/offline_template_full.yaml +++ b/src/inference_endpoint/config/templates/offline_template_full.yaml @@ -43,7 +43,7 @@ datasets: # Dataset configs system: system_prompt generate_params: null # Dataset-specific parameters passed to the generate() method accuracy_config: # Accuracy evaluation settings - eval_method: pass_at_1 # Scorer method | options: pass_at_1, string_match, rouge, code_bench_scorer, shopify_category_f1, agentic_inference_inline, vbench, bfcl_v4, legacy_mlperf_deepseek_r1 + eval_method: pass_at_1 # Scorer method | options: pass_at_1, string_match, rouge, code_bench_scorer, shopify_category_f1, agentic_inference_inline, vbench, bfcl_v4, legacy_mlperf_deepseek_r1, swe_bench_scorer ground_truth: ground_truth # Ground truth column name extractor: boxed_math_extractor # Answer extractor (abcd_extractor, boxed_math_extractor, identity_extractor, python_code_extractor) num_repeats: 1 # Repeat dataset N times for evaluation diff --git a/src/inference_endpoint/config/templates/online_template_full.yaml b/src/inference_endpoint/config/templates/online_template_full.yaml index c29abc23f..003f0685e 100644 --- a/src/inference_endpoint/config/templates/online_template_full.yaml +++ b/src/inference_endpoint/config/templates/online_template_full.yaml @@ -43,7 +43,7 @@ datasets: # Dataset configs system: system_prompt generate_params: null # Dataset-specific parameters passed to the generate() method accuracy_config: # Accuracy evaluation settings - eval_method: pass_at_1 # Scorer method | options: pass_at_1, string_match, rouge, code_bench_scorer, shopify_category_f1, agentic_inference_inline, vbench, bfcl_v4, legacy_mlperf_deepseek_r1 + eval_method: pass_at_1 # Scorer method | options: pass_at_1, string_match, rouge, code_bench_scorer, shopify_category_f1, agentic_inference_inline, vbench, bfcl_v4, legacy_mlperf_deepseek_r1, swe_bench_scorer ground_truth: ground_truth # Ground truth column name extractor: boxed_math_extractor # Answer extractor (abcd_extractor, boxed_math_extractor, identity_extractor, python_code_extractor) num_repeats: 1 # Repeat dataset N times for evaluation diff --git a/src/inference_endpoint/dataset_manager/__init__.py b/src/inference_endpoint/dataset_manager/__init__.py index 213ee2e1b..98c96f868 100644 --- a/src/inference_endpoint/dataset_manager/__init__.py +++ b/src/inference_endpoint/dataset_manager/__init__.py @@ -38,6 +38,7 @@ ShopifyProductCatalogue, ShopifyProductCatalogue8k, ) +from .predefined.swe_bench import SWEBench from .transforms import ( AddStaticColumns, ColumnFilter, @@ -70,5 +71,6 @@ "RandomDataset", "ShopifyProductCatalogue", "ShopifyProductCatalogue8k", + "SWEBench", "AgenticInferenceDataset", ] diff --git a/src/inference_endpoint/dataset_manager/dataset.py b/src/inference_endpoint/dataset_manager/dataset.py index bd259d7a1..59e218115 100644 --- a/src/inference_endpoint/dataset_manager/dataset.py +++ b/src/inference_endpoint/dataset_manager/dataset.py @@ -281,6 +281,10 @@ class Dataset: DATASET_ID: ClassVar[str] """The unique identifier for the dataset. Automatically set by __init_subclass__.""" + ACCURACY_ONLY: ClassVar[bool] = False + """If True, this dataset may only be used as an accuracy dataset (type: accuracy). + Using it as a performance dataset raises InputValidationError at load time.""" + def __init_subclass__( cls, dataset_id: str | None = None, diff --git a/src/inference_endpoint/dataset_manager/predefined/swe_bench/__init__.py b/src/inference_endpoint/dataset_manager/predefined/swe_bench/__init__.py new file mode 100644 index 000000000..210c74234 --- /dev/null +++ b/src/inference_endpoint/dataset_manager/predefined/swe_bench/__init__.py @@ -0,0 +1,132 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from logging import getLogger +from pathlib import Path + +import pandas as pd + +from ...dataset import Dataset, load_from_huggingface + +logger = getLogger(__name__) + +_REPO_MAP = { + "verified": "princeton-nlp/SWE-bench_Verified", + "lite": "princeton-nlp/SWE-bench_Lite", +} + + +class SWEBench( + Dataset, + dataset_id="swe_bench", +): + """SWE-bench: Software Engineering Benchmark for LLM agents. + + Loads instance IDs and problem statements from the SWE-bench Verified or + Lite subset. Used as the accuracy dataset for the swe_bench_scorer, which + runs mini-swe-agent against a live endpoint and grades patches with the + SWE-bench evaluation harness. + + The ``instance_id`` column identifies which instances mini-swe-agent will + evaluate. The endpoint phase is skipped entirely for this scorer + (``SKIP_ENDPOINT_PHASE=True``); ``SWEBenchScorer`` drives the agent + subprocess directly against the configured endpoint. + + Using this dataset as a performance dataset (type: performance) is not + meaningful — problem statements sent directly to the model without an + agent framework don't reflect real SWE-bench usage. Use a different + dataset (e.g. ``random``) for the performance phase. + """ + + ACCURACY_ONLY = True + COLUMN_NAMES = ["instance_id", "prompt"] + + @classmethod + def hf_dataset_name(cls, subset: str) -> str: + hf_path = _REPO_MAP.get(subset) + if hf_path is None: + raise ValueError( + f"Unknown SWE-bench subset {subset!r}; choose from: {list(_REPO_MAP)}" + ) + return hf_path + + @classmethod + def generate( + cls, + datasets_dir: Path, + subset: str = "verified", + split: str = "test", + force: bool = False, + ) -> pd.DataFrame: + """Download and cache the SWE-bench dataset from HuggingFace. + + Args: + datasets_dir: Root cache directory. Parquet is written under + ``datasets_dir/swe_bench/{subset}/{split}/``. + subset: ``"verified"`` (500 instances) or ``"lite"`` (300 instances). + split: HuggingFace split to load. Defaults to ``"test"``. + force: Re-download even if the local parquet cache exists. + + Returns: + DataFrame with columns ``instance_id`` and ``prompt``. + """ + hf_path = cls.hf_dataset_name(subset) + + cache_suffix = f"swe_bench_{subset}_{split}" + dst_path = ( + datasets_dir / "swe_bench" / subset / split / f"{cache_suffix}.parquet" + ) + if dst_path.exists() and not force: + logger.info( + "Loading SWE-bench %s/%s from cache: %s", subset, split, dst_path + ) + try: + return pd.read_parquet(dst_path) + except Exception as e: + raise RuntimeError( + f"Cached SWE-bench parquet at {dst_path} appears corrupt ({e}). " + "Delete it or pass force=True to re-download." + ) from e + + try: + df = load_from_huggingface( + hf_path, + split=split, + cache_dir=datasets_dir / "hf_cache" / cache_suffix, + ) + except Exception as e: + logger.error( + "Error loading SWE-bench %s/%s from HuggingFace: %s", + subset, + split, + e, + ) + raise + + result = ( + df[["instance_id", "problem_statement"]] + .rename(columns={"problem_statement": "prompt"}) + .reset_index(drop=True) + ) + dst_path.parent.mkdir(parents=True, exist_ok=True) + result.to_parquet(dst_path) + logger.info( + "Saved %d SWE-bench %s/%s instances to %s", + len(result), + subset, + split, + dst_path, + ) + return result diff --git a/src/inference_endpoint/evaluation/scoring.py b/src/inference_endpoint/evaluation/scoring.py index bb756e45f..e88a9f74e 100644 --- a/src/inference_endpoint/evaluation/scoring.py +++ b/src/inference_endpoint/evaluation/scoring.py @@ -23,17 +23,22 @@ import subprocess import sys import tempfile +import time import uuid from abc import ABC, abstractmethod from collections import Counter, defaultdict from collections.abc import Iterator from pathlib import Path from typing import Any, ClassVar +from urllib import error as urllib_error +from urllib import request as urllib_request +from urllib.parse import urljoin, urlparse import msgspec import msgspec.json import numpy as np import pandas as pd +import yaml from pydantic import ValidationError from tqdm import tqdm @@ -54,6 +59,8 @@ from ..dataset_manager.agentic_inference_dataset import AgenticInferenceDataset from ..dataset_manager.dataset import Dataset from ..dataset_manager.predefined.shopify_product_catalogue import ProductMetadata +from ..dataset_manager.predefined.swe_bench import SWEBench +from ..exceptions import SetupError from .accuracy_results import build_breakdown from .extractor import ( Extractor, @@ -72,6 +79,7 @@ class Scorer(ABC): PREDEFINED: ClassVar[dict[str, type["Scorer"]]] = {} SCORER_ID: ClassVar[str] REQUIRES_EXTRACTOR: ClassVar[bool] = True + SKIP_ENDPOINT_PHASE: ClassVar[bool] = False def __init_subclass__( cls, @@ -111,6 +119,20 @@ def available_scorers(cls) -> list[str]: """Return the list of registered scorer names.""" return list(Scorer.PREDEFINED.keys()) + @classmethod + def dataset_loader_kwargs(cls, extras: dict[str, Any]) -> dict[str, Any]: + return {} + + @classmethod + def external_sample_count(cls, extras: dict[str, Any]) -> int | None: + return None + + @classmethod + def preflight( + cls, extras: dict[str, Any], *, loaded_sample_count: int | None = None + ) -> None: + return None + def __init__( self, dataset_name: str, @@ -1700,6 +1722,700 @@ def score(self) -> tuple[float | None, int]: return mean_score, n_repeats +class SWEBenchScorer(Scorer, scorer_id="swe_bench_scorer"): + """SWE-bench accuracy scorer backed by a remote SWE-bench service.""" + + REQUIRES_EXTRACTOR: ClassVar[bool] = False + SKIP_ENDPOINT_PHASE: ClassVar[bool] = True + DEFAULT_SUBSET: ClassVar[str] = "verified" + DEFAULT_SPLIT: ClassVar[str] = "test" + DEFAULT_NUM_INSTANCES: ClassVar[int] = 100 + DEFAULT_WORKERS: ClassVar[int] = 10 + DEFAULT_MAX_EVAL_WORKERS: ClassVar[int] = 10 + DEFAULT_SERVICE_TIMEOUT_S: ClassVar[int] = 24 * 60 * 60 + DEFAULT_POLL_INTERVAL_S: ClassVar[float] = 5.0 + SERVICE_API_VERSION: ClassVar[str] = "v1" + REQUIRED_SERVICE_CAPABILITIES: ClassVar[set[str]] = { + "swebench.run", + "swebench.cancel", + "artifacts.download", + } + SAFE_ARTIFACT_NAMES: ClassVar[set[str]] = { + "preds.json", + "swe_bench_agent.log", + "swe_bench_eval.log", + "swe_bench_results.json", + "status.json", + } + TOOLCALL_PATCH_EXTRA: ClassVar[str] = "enable_swebench_toolcall_patch" + SERVICE_TEMPLATES: ClassVar[set[str]] = {"default", "qwen_tools"} + + def __init__( + self, + dataset_name: str, + dataset: Dataset, + report_dir: os.PathLike, + extractor: type[Extractor] | None = None, + ground_truth_column: str | None = "instance_id", + swebench_service_url: str | None = None, + swebench_service_auth_token: str | None = None, + subset: str = "verified", + split: str = "test", + num_instances: int = 100, + workers: int = 10, + max_eval_workers: int = 10, + enable_swebench_toolcall_patch: bool = False, + swebench_template: str | None = None, + service_timeout_s: int | None = None, + poll_interval_s: float | None = None, + ): + ground_truth_column = ground_truth_column or "instance_id" + super().__init__( + dataset_name=dataset_name, + dataset=dataset, + report_dir=report_dir, + extractor=extractor, + ground_truth_column=ground_truth_column, + ) + self.report_dir = self.report_dir.resolve() + options = self._resolve_options( + { + "swebench_service_url": swebench_service_url, + "swebench_service_auth_token": swebench_service_auth_token, + "subset": subset, + "split": split, + "num_instances": num_instances, + "workers": workers, + "max_eval_workers": max_eval_workers, + self.TOOLCALL_PATCH_EXTRA: enable_swebench_toolcall_patch, + "swebench_template": swebench_template, + "service_timeout_s": service_timeout_s, + "poll_interval_s": poll_interval_s, + } + ) + self.swebench_service_url = options["swebench_service_url"] + self.swebench_service_auth_token = options["swebench_service_auth_token"] + self.subset = options["subset"] + self.split = options["split"] + self.num_instances = options["num_instances"] + self.workers = options["workers"] + self.max_eval_workers = options["max_eval_workers"] + self.enable_swebench_toolcall_patch = options[self.TOOLCALL_PATCH_EXTRA] + self.swebench_template = options["swebench_template"] + self.service_timeout_s = options["service_timeout_s"] + self.poll_interval_s = options["poll_interval_s"] + + @classmethod + def _normalize_service_url(cls, value: Any) -> str: + if value is None or str(value).strip() == "": + raise SetupError( + "accuracy_config.extras.swebench_service_url is required for " + "swe_bench_scorer. Start the SWE-bench service and pass its URL." + ) + return str(value).strip().rstrip("/") + "/" + + @classmethod + def _http_json( + cls, + url: str, + *, + method: str = "GET", + payload: dict[str, Any] | None = None, + timeout_s: float = 30.0, + auth_token: str | None = None, + ) -> dict[str, Any]: + data = None + headers = {"Accept": "application/json"} + if auth_token: + headers["Authorization"] = f"Bearer {auth_token}" + if payload is not None: + data = msgspec.json.encode(payload) + headers["Content-Type"] = "application/json" + req = urllib_request.Request(url, data=data, headers=headers, method=method) + try: + with urllib_request.urlopen(req, timeout=timeout_s) as resp: + body = resp.read() + except urllib_error.HTTPError as exc: + detail = exc.read().decode("utf-8", errors="replace") + raise SetupError( + f"SWE-bench service request failed: {url} returned HTTP " + f"{exc.code}: {detail}" + ) from exc + except urllib_error.URLError as exc: + raise SetupError( + f"SWE-bench service is unreachable at {url}: {exc.reason}" + ) from exc + try: + decoded = msgspec.json.decode(body, type=dict) + except msgspec.DecodeError as exc: + raise SetupError( + f"SWE-bench service returned invalid JSON from {url}" + ) from exc + return decoded + + @classmethod + def _check_health( + cls, service_url: str, auth_token: str | None = None + ) -> dict[str, Any]: + health = cls._http_json( + urljoin(service_url, "health"), + timeout_s=10.0, + auth_token=auth_token, + ) + api_version = health.get("api_version") + if api_version != cls.SERVICE_API_VERSION: + raise SetupError( + "SWE-bench service API version mismatch: expected " + f"{cls.SERVICE_API_VERSION!r}, got {api_version!r}" + ) + capabilities = set(health.get("capabilities") or []) + missing = cls.REQUIRED_SERVICE_CAPABILITIES - capabilities + if missing: + raise SetupError( + "SWE-bench service is missing required capabilities: " + + ", ".join(sorted(missing)) + ) + return health + + @classmethod + def _download_artifact( + cls, + service_url: str, + artifact: dict[str, Any], + report_dir: Path, + run_id: str, + auth_token: str | None = None, + ) -> None: + name = str(artifact.get("name") or "") + href = str(artifact.get("url") or "") + if name not in cls.SAFE_ARTIFACT_NAMES or not href: + return + parsed = urlparse(href) + expected_path = f"/v1/runs/{run_id}/artifacts/{name}" + if ( + parsed.scheme + or parsed.netloc + or parsed.params + or parsed.query + or parsed.fragment + or parsed.path != expected_path + ): + logger.warning("Ignoring unsafe SWE-bench artifact URL for %s", name) + return + target = report_dir / name + url = urljoin(service_url, href.lstrip("/")) + req = urllib_request.Request( + url, headers={"Accept": "application/octet-stream"} + ) + if auth_token: + req.add_header("Authorization", f"Bearer {auth_token}") + try: + with urllib_request.urlopen(req, timeout=60.0) as resp: + target.write_bytes(resp.read()) + except Exception: + logger.warning( + "Could not download SWE-bench artifact %s", name, exc_info=True + ) + + @classmethod + def _download_artifacts( + cls, + service_url: str, + status: dict[str, Any], + report_dir: Path, + auth_token: str | None = None, + ) -> None: + run_id = str(status.get("run_id") or "") + if not run_id: + return + artifacts = status.get("artifacts") or [] + if isinstance(artifacts, dict): + artifacts = [ + {"name": name, "url": url} + for name, url in artifacts.items() + if isinstance(name, str) + ] + if not isinstance(artifacts, list): + return + for artifact in artifacts: + if isinstance(artifact, dict): + cls._download_artifact( + service_url, artifact, report_dir, run_id, auth_token + ) + + @classmethod + def _cancel_service_run( + cls, service_url: str, run_id: str, auth_token: str | None = None + ) -> None: + try: + cls._http_json( + urljoin(service_url, f"v1/runs/{run_id}/cancel"), + method="POST", + timeout_s=10.0, + auth_token=auth_token, + ) + except SetupError: + logger.warning("Could not cancel SWE-bench service run %s", run_id) + + @staticmethod + def _write_service_status(report_dir: Path, status: dict[str, Any]) -> None: + (report_dir / "swe_bench_service_status.json").write_bytes( + msgspec.json.encode(status) + ) + + @staticmethod + def _progress_int(status: dict[str, Any], key: str) -> int | None: + value = status.get(key) + if value is None: + return None + try: + parsed = int(value) + except (TypeError, ValueError): + return None + return parsed if parsed >= 0 else None + + @classmethod + def _update_progress_bars( + cls, status: dict[str, Any], state: dict[str, Any] + ) -> None: + phase = str(status.get("phase") or "") + agent_total = cls._progress_int(status, "agent_total") + agent_completed = cls._progress_int(status, "agent_completed") + eval_total = cls._progress_int(status, "eval_total") + eval_completed = cls._progress_int(status, "eval_completed") + if ( + not phase + and agent_total is None + and agent_completed is None + and eval_total is None + and eval_completed is None + ): + return + + agent_bar = state.get("agent_bar") + if agent_bar is None and agent_total is not None and agent_total > 0: + agent_bar = tqdm( + total=agent_total, + desc="SWE-bench agent", + unit="inst", + ) + state["agent_bar"] = agent_bar + state["agent_completed"] = 0 + if agent_bar is not None: + if agent_total is not None and agent_total > (agent_bar.total or 0): + agent_bar.total = agent_total + agent_bar.refresh() + if agent_completed is not None: + previous = int(state.get("agent_completed") or 0) + current = max(previous, agent_completed) + if current > previous: + agent_bar.update(current - previous) + state["agent_completed"] = current + + eval_bar = state.get("eval_bar") + should_open_eval = ( + phase in {"eval", "succeeded"} and eval_total is not None and eval_total > 0 + ) + if eval_bar is None and should_open_eval: + eval_bar = tqdm( + total=eval_total, + desc="SWE-bench eval", + unit="inst", + ) + state["eval_bar"] = eval_bar + state["eval_completed"] = 0 + if eval_bar is not None: + if eval_total is not None and eval_total > (eval_bar.total or 0): + eval_bar.total = eval_total + eval_bar.refresh() + if eval_completed is not None: + previous = int(state.get("eval_completed") or 0) + current = max(previous, eval_completed) + if current > previous: + eval_bar.update(current - previous) + state["eval_completed"] = current + + @staticmethod + def _close_progress_bars(state: dict[str, Any]) -> None: + for key in ("eval_bar", "agent_bar"): + bar = state.get(key) + if bar is not None: + bar.close() + + @classmethod + def _get_extra_int( + cls, extras: dict[str, Any], key: str, *, default: int, min_value: int = 0 + ) -> int: + value = extras.get(key) + if value is None: + value = default + try: + parsed = int(value) + except (TypeError, ValueError) as exc: + raise SetupError( + f"accuracy_config.extras.{key} must be an integer; got {value!r}" + ) from exc + if parsed < min_value: + raise SetupError( + f"accuracy_config.extras.{key} must be >= {min_value}; got {parsed}" + ) + return parsed + + @staticmethod + def _validate_max_eval_workers(value: Any) -> int: + try: + parsed = int(value) + except (TypeError, ValueError) as exc: + raise ValueError( + f"max_eval_workers must be an integer; got {value!r}" + ) from exc + if parsed < 1: + raise ValueError(f"max_eval_workers must be >= 1; got {parsed}") + return parsed + + @classmethod + def _get_extra_bool( + cls, extras: dict[str, Any], key: str, *, default: bool = False + ) -> bool: + value = extras.get(key) + if value is None: + value = default + if isinstance(value, bool): + return value + if isinstance(value, int) and value in (0, 1): + return bool(value) + if isinstance(value, str): + normalized = value.strip().lower() + if normalized in {"1", "true", "yes", "on"}: + return True + if normalized in {"0", "false", "no", "off"}: + return False + raise SetupError( + f"accuracy_config.extras.{key} must be a boolean; got {value!r}" + ) + + @classmethod + def _get_extra_float( + cls, extras: dict[str, Any], key: str, *, default: float, min_value: float = 0 + ) -> float: + value = extras.get(key) + if value is None: + value = default + try: + parsed = float(value) + except (TypeError, ValueError) as exc: + raise SetupError( + f"accuracy_config.extras.{key} must be numeric; got {value!r}" + ) from exc + if parsed <= min_value: + raise SetupError( + f"accuracy_config.extras.{key} must be > {min_value:g}; got {parsed}" + ) + return parsed + + @classmethod + def _resolve_dataset_options(cls, extras: dict[str, Any]) -> dict[str, str]: + subset = str(extras.get("subset", cls.DEFAULT_SUBSET)) + SWEBench.hf_dataset_name(subset) + return { + "subset": subset, + "split": str(extras.get("split", cls.DEFAULT_SPLIT)), + } + + @classmethod + def _resolve_service_template(cls, extras: dict[str, Any]) -> str: + raw = extras.get("swebench_template") + if raw is None: + raw = ( + "qwen_tools" + if cls._get_extra_bool(extras, cls.TOOLCALL_PATCH_EXTRA) + else "default" + ) + template = str(raw) + if template not in cls.SERVICE_TEMPLATES: + raise SetupError( + "accuracy_config.extras.swebench_template must be one of " + f"{sorted(cls.SERVICE_TEMPLATES)}; got {template!r}" + ) + return template + + @classmethod + def _resolve_options(cls, extras: dict[str, Any]) -> dict[str, Any]: + options: dict[str, Any] = cls._resolve_dataset_options(extras) + options["swebench_service_url"] = cls._normalize_service_url( + extras.get("swebench_service_url") + ) + auth_token = extras.get("swebench_service_auth_token") + options["swebench_service_auth_token"] = ( + str(auth_token) if auth_token not in (None, "") else None + ) + options["num_instances"] = cls._get_extra_int( + extras, + "num_instances", + default=cls.DEFAULT_NUM_INSTANCES, + min_value=1, + ) + options["workers"] = cls._get_extra_int( + extras, + "workers", + default=cls.DEFAULT_WORKERS, + min_value=1, + ) + options["max_eval_workers"] = cls._get_extra_int( + extras, + "max_eval_workers", + default=cls.DEFAULT_MAX_EVAL_WORKERS, + min_value=1, + ) + options[cls.TOOLCALL_PATCH_EXTRA] = cls._get_extra_bool( + extras, + cls.TOOLCALL_PATCH_EXTRA, + ) + options["swebench_template"] = cls._resolve_service_template(extras) + options["service_timeout_s"] = cls._get_extra_int( + extras, + "service_timeout_s", + default=cls.DEFAULT_SERVICE_TIMEOUT_S, + min_value=1, + ) + options["poll_interval_s"] = cls._get_extra_float( + extras, + "poll_interval_s", + default=cls.DEFAULT_POLL_INTERVAL_S, + min_value=0, + ) + return options + + @staticmethod + def _generation_params(model_params: dict[str, Any]) -> dict[str, Any]: + fields = ( + "temperature", + "top_p", + "top_k", + "repetition_penalty", + "presence_penalty", + "frequency_penalty", + "max_new_tokens", + "chat_template_kwargs", + ) + return {field: model_params[field] for field in fields if field in model_params} + + @classmethod + def dataset_loader_kwargs(cls, extras: dict[str, Any]) -> dict[str, Any]: + return cls._resolve_dataset_options(extras) + + @classmethod + def external_sample_count(cls, extras: dict[str, Any]) -> int | None: + raw = extras.get("num_instances", cls.DEFAULT_NUM_INSTANCES) + try: + parsed = int(raw) + except (TypeError, ValueError): + return None + return parsed if parsed > 0 else None + + @classmethod + def preflight( + cls, extras: dict[str, Any], *, loaded_sample_count: int | None = None + ) -> None: + """Check the SWE-bench service before the benchmark starts.""" + try: + options = cls._resolve_options(extras) + except ValueError as exc: + raise SetupError(str(exc)) from exc + cls._check_health( + options["swebench_service_url"], + options["swebench_service_auth_token"], + ) + + def score_single_sample(self, value: str, ground_truth: str) -> float: + raise RuntimeError( + "SWEBenchScorer uses service evaluation; call score() instead." + ) + + def score(self) -> tuple[float | None, int]: + """Submit a SWE-bench service run. Returns (resolved_rate, 1).""" + self.complete = True + config_path = self.report_dir / "config.yaml" + if not config_path.exists(): + raise FileNotFoundError( + f"config.yaml not found at {config_path}. " + "SWEBenchScorer.score() must be called from within a benchmark run " + "that has already written its config, or the path must be pre-populated." + ) + with config_path.open() as f: + benchmark_cfg = yaml.safe_load(f) + if not isinstance(benchmark_cfg, dict): + raise ValueError( + f"benchmark config at {config_path} must be a YAML mapping" + ) + + model_params = benchmark_cfg.get("model_params") or {} + model_name = model_params.get("name") + if not model_name: + raise ValueError( + "model_params.name is required in the benchmark config but is missing or empty" + ) + if self.dataset.dataframe is None: + raise RuntimeError( + "SWEBench dataset must be loaded before scoring; call dataset.load() first." + ) + + n_rows = len(self.dataset.dataframe) + if self.num_instances > n_rows: + logger.warning( + "num_instances=%d exceeds dataset size %d; evaluating %d instances", + self.num_instances, + n_rows, + n_rows, + ) + total_instances = min(self.num_instances, n_rows) + evaluated_instance_ids = [ + str(instance_id) + for instance_id in self.dataset.dataframe.iloc[:total_instances][ + self.ground_truth_column + ].tolist() + ] + if not evaluated_instance_ids: + logger.warning("SWE-bench: no evaluated instances; returning None score") + self.complete = False + return None, 1 + + endpoint_config = benchmark_cfg.get("endpoint_config") or {} + endpoint_urls = endpoint_config.get("endpoints") or [] + if len(endpoint_urls) != 1: + raise SetupError( + "SWE-bench service mode supports exactly one endpoint URL; " + f"got {len(endpoint_urls)}." + ) + payload: dict[str, Any] = { + "model_name": model_name, + "endpoint_urls": endpoint_urls, + "endpoint_api_key": endpoint_config.get("api_key"), + "generation_params": self._generation_params(model_params), + "subset": self.subset, + "split": self.split, + "num_instances": total_instances, + "workers": self.workers, + "max_eval_workers": self.max_eval_workers, + "evaluated_instance_ids": evaluated_instance_ids, + self.TOOLCALL_PATCH_EXTRA: self.enable_swebench_toolcall_patch, + "template": self.swebench_template, + } + + run_id = "" + progress_state: dict[str, Any] = {} + try: + submitted = type(self)._http_json( + urljoin(self.swebench_service_url, "v1/runs"), + method="POST", + payload=payload, + timeout_s=30.0, + auth_token=self.swebench_service_auth_token, + ) + run_id = str(submitted.get("run_id") or "") + if not run_id: + raise SetupError("SWE-bench service did not return run_id") + type(self)._update_progress_bars(submitted, progress_state) + + deadline = time.monotonic() + self.service_timeout_s + status = submitted + while status.get("status") not in {"succeeded", "failed", "cancelled"}: + if time.monotonic() >= deadline: + raise SetupError( + f"Timed out waiting for SWE-bench service run {run_id}" + ) + time.sleep(self.poll_interval_s) + status = type(self)._http_json( + urljoin(self.swebench_service_url, f"v1/runs/{run_id}"), + timeout_s=30.0, + auth_token=self.swebench_service_auth_token, + ) + type(self)._update_progress_bars(status, progress_state) + except (KeyboardInterrupt, SystemExit): + if run_id: + type(self)._cancel_service_run( + self.swebench_service_url, + run_id, + self.swebench_service_auth_token, + ) + raise + except SetupError: + if run_id: + type(self)._cancel_service_run( + self.swebench_service_url, + run_id, + self.swebench_service_auth_token, + ) + logger.error("SWE-bench service run failed", exc_info=True) + self.complete = False + return None, 1 + finally: + type(self)._close_progress_bars(progress_state) + + type(self)._write_service_status(self.report_dir, status) + type(self)._download_artifacts( + self.swebench_service_url, + status, + self.report_dir, + self.swebench_service_auth_token, + ) + if status.get("status") != "succeeded": + logger.error( + "SWE-bench service run %s ended with status %s", + run_id, + status.get("status"), + ) + self.complete = False + return None, 1 + + result = status.get("result") + result_path = self.report_dir / "swe_bench_results.json" + if result is None and result_path.exists(): + try: + result = msgspec.json.decode(result_path.read_bytes(), type=dict) + except msgspec.DecodeError: + self.complete = False + return None, 1 + if not isinstance(result, dict): + logger.error("SWE-bench service run %s did not return a result", run_id) + self.complete = False + return None, 1 + if not result_path.exists(): + result_path.write_bytes(msgspec.json.encode(result)) + + submitted_count = result.get("submitted_instances") or 0 + resolved = result.get("resolved_instances") or 0 + if submitted_count == 0: + logger.warning("SWE-bench: submitted_instances=0; returning None score") + self.complete = False + return None, 1 + + denominator = len(evaluated_instance_ids) + if denominator == 0: + logger.warning( + "SWE-bench: evaluated instance count is 0; returning None score" + ) + self.complete = False + return None, 1 + if submitted_count != denominator: + logger.warning( + "SWE-bench: service submitted %d / %d evaluated instances; " + "marking score incomplete", + submitted_count, + denominator, + ) + self.complete = False + + resolved_rate = resolved / denominator + logger.info( + "SWE-bench: resolved %d / %d evaluated (%.1f%%)", + resolved, + denominator, + resolved_rate * 100, + ) + return resolved_rate, 1 + + class LegacyMLPerfDeepSeekR1Scorer(Scorer, scorer_id="legacy_mlperf_deepseek_r1"): """MLPerf DeepSeek-R1 combined-subset accuracy scorer. diff --git a/src/inference_endpoint/evaluation/swebench_service/README.md b/src/inference_endpoint/evaluation/swebench_service/README.md new file mode 100644 index 000000000..14f22cc05 --- /dev/null +++ b/src/inference_endpoint/evaluation/swebench_service/README.md @@ -0,0 +1,32 @@ +# SWE-bench Service + +Runs mini-swe-agent and the SWE-bench harness on a host with Docker. The +benchmark client only needs this service URL, but the service is trusted +infrastructure: it receives one endpoint URL and optional endpoint credentials, runs +Docker-backed evaluations, and serves run artifacts. + +```bash +uv run --project src/inference_endpoint/evaluation/swebench_service \ + python -m swebench_service --host 0.0.0.0 --port 18080 +``` + +The endpoint URL in the benchmark config must be reachable from the service +host. Service mode supports exactly one endpoint URL and follows the +LiveCodeBench-style external-service convention for heavyweight evaluation work. +Docker is required only on the service host. + +For non-loopback deployments, bind only on a private network or set +`--auth-token TOKEN` and configure the client with: + +```yaml +accuracy_config: + extras: + swebench_service_url: http://swebench-host:18080 + swebench_service_auth_token: TOKEN +``` + +The service selects templates from its packaged allowlist. Use +`swebench_template: qwen_tools` with +`enable_swebench_toolcall_patch: true` for Qwen tool-call runs; otherwise omit +the template option. Completed run metadata and artifacts are retained up to +`--max-stored-runs` runs. diff --git a/src/inference_endpoint/evaluation/swebench_service/pyproject.toml b/src/inference_endpoint/evaluation/swebench_service/pyproject.toml new file mode 100644 index 000000000..a214290e4 --- /dev/null +++ b/src/inference_endpoint/evaluation/swebench_service/pyproject.toml @@ -0,0 +1,24 @@ +[build-system] +requires = ["uv_build==0.7.6"] +build-backend = "uv_build" + +[tool.uv.build-backend] +module-root = "." + +[project] +name = "swebench-service" +version = "0.1.0" +description = "Native SWE-bench service for MLPerf endpoint accuracy runs." +requires-python = ">=3.12,<3.13" +dependencies = [ + "aiohttp==3.14.1", + "mini-swe-agent==2.3.0", + "swebench==4.1.0", + "litellm==1.91.0", + "msgspec==0.20.0", + "pydantic==2.12.5", + "pyyaml==6.0.3", +] + +[tool.uv] +package = true diff --git a/src/inference_endpoint/evaluation/swebench_service/swebench_service/__init__.py b/src/inference_endpoint/evaluation/swebench_service/swebench_service/__init__.py new file mode 100644 index 000000000..6873cbd37 --- /dev/null +++ b/src/inference_endpoint/evaluation/swebench_service/swebench_service/__init__.py @@ -0,0 +1,24 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Native SWE-bench service.""" + +API_VERSION = "v1" +CAPABILITIES = [ + "swebench.run", + "swebench.cancel", + "artifacts.download", + "swebench.progress", +] diff --git a/src/inference_endpoint/evaluation/swebench_service/swebench_service/__main__.py b/src/inference_endpoint/evaluation/swebench_service/swebench_service/__main__.py new file mode 100644 index 000000000..2344c58d5 --- /dev/null +++ b/src/inference_endpoint/evaluation/swebench_service/swebench_service/__main__.py @@ -0,0 +1,51 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import argparse +from pathlib import Path + +from aiohttp import web + +from .config import ServiceConfig +from .server import create_app + + +def main() -> None: + parser = argparse.ArgumentParser(description="Run the SWE-bench service") + parser.add_argument("--host", default="127.0.0.1") + parser.add_argument("--port", type=int, default=18080) + parser.add_argument("--artifact-root", default="swebench_service_artifacts") + parser.add_argument("--max-concurrent-runs", type=int, default=1) + parser.add_argument("--subprocess-timeout-s", type=int, default=24 * 60 * 60) + parser.add_argument("--auth-token") + parser.add_argument("--max-stored-runs", type=int, default=100) + args = parser.parse_args() + + config = ServiceConfig( + host=args.host, + port=args.port, + artifact_root=Path(args.artifact_root), + max_concurrent_runs=args.max_concurrent_runs, + subprocess_timeout_s=args.subprocess_timeout_s, + auth_token=args.auth_token, + max_stored_runs=args.max_stored_runs, + ) + web.run_app(create_app(config), host=config.host, port=config.port) + + +if __name__ == "__main__": + main() diff --git a/src/inference_endpoint/evaluation/swebench_service/swebench_service/artifacts.py b/src/inference_endpoint/evaluation/swebench_service/swebench_service/artifacts.py new file mode 100644 index 000000000..45ce09bc4 --- /dev/null +++ b/src/inference_endpoint/evaluation/swebench_service/swebench_service/artifacts.py @@ -0,0 +1,103 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import re +from pathlib import Path +from typing import Any + +SECRET_KEY_NAMES = { + "api-key", + "api_key", + "authorization", + "access-token", + "access_token", + "endpoint-api-key", + "endpoint_api_key", + "password", + "secret", + "secret-key", + "secret_key", + "token", + "x-api-key", + "x_api_key", +} +SECRET_TEXT_PATTERNS = ( + re.compile(r"(?i)(authorization:\s*(?:bearer|basic)\s+)[^\s,;]+"), + re.compile(r"(?i)((?:api[_-]?key|access[_-]?token|token|password)=)[^&\s]+"), + re.compile(r"(://[^:/\s]+:)[^@\s/]+(@)"), +) +SAFE_ARTIFACT_NAMES = { + "preds.json", + "swe_bench_agent.log", + "swe_bench_eval.log", + "swe_bench_results.json", + "status.json", +} + + +def _is_secret_key(key: Any) -> bool: + normalized = str(key).strip().lower() + if normalized in SECRET_KEY_NAMES: + return True + compact = normalized.replace("-", "_") + return ( + compact.endswith("_key") + or compact.endswith("_token") + or "api_key" in compact + or "access_token" in compact + or "secret" in compact + or "password" in compact + ) + + +def redact_text(text: str, secret_values: set[str] | None = None) -> str: + redacted = text + for secret in secret_values or set(): + if len(secret) >= 4: + redacted = redacted.replace(secret, "") + for pattern in SECRET_TEXT_PATTERNS: + if pattern.pattern.startswith("(://"): + redacted = pattern.sub(r"\1\2", redacted) + else: + redacted = pattern.sub(r"\1", redacted) + return redacted + + +def redact_secrets(value: Any, *, secret_values: set[str] | None = None) -> Any: + if isinstance(value, dict): + redacted = {} + for key, val in value.items(): + if _is_secret_key(key): + redacted[key] = "" + else: + redacted[key] = redact_secrets(val, secret_values=secret_values) + return redacted + if isinstance(value, list): + return [redact_secrets(item, secret_values=secret_values) for item in value] + if isinstance(value, str): + return redact_text(value, secret_values) + return value + + +def resolve_artifact(run_dir: Path, name: str) -> Path: + if name not in SAFE_ARTIFACT_NAMES or "/" in name or "\\" in name: + raise FileNotFoundError(name) + path = (run_dir / name).resolve() + root = run_dir.resolve() + if path.parent != root or not path.is_file(): + raise FileNotFoundError(name) + return path diff --git a/src/inference_endpoint/evaluation/swebench_service/swebench_service/config.py b/src/inference_endpoint/evaluation/swebench_service/swebench_service/config.py new file mode 100644 index 000000000..6c11c1d48 --- /dev/null +++ b/src/inference_endpoint/evaluation/swebench_service/swebench_service/config.py @@ -0,0 +1,30 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +from dataclasses import dataclass +from pathlib import Path + + +@dataclass(frozen=True) +class ServiceConfig: + host: str = "127.0.0.1" + port: int = 18080 + artifact_root: Path = Path("swebench_service_artifacts") + max_concurrent_runs: int = 1 + subprocess_timeout_s: int = 24 * 60 * 60 + auth_token: str | None = None + max_stored_runs: int = 100 diff --git a/src/inference_endpoint/evaluation/swebench_service/swebench_service/runner.py b/src/inference_endpoint/evaluation/swebench_service/swebench_service/runner.py new file mode 100644 index 000000000..96bb4eaab --- /dev/null +++ b/src/inference_endpoint/evaluation/swebench_service/swebench_service/runner.py @@ -0,0 +1,501 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import os +import re +import shutil +import signal +import subprocess +import sys +import tempfile +import threading +import time +import uuid +from pathlib import Path +from typing import Any +from urllib.parse import urlparse, urlunparse + +import msgspec.json +import yaml + +from .artifacts import redact_secrets +from .schemas import RunRequest, TemplateName + + +class RunnerError(RuntimeError): + pass + + +class RunCancelled(RunnerError): + pass + + +class CancellationToken: + def __init__(self) -> None: + self._event = threading.Event() + self._lock = threading.Lock() + self._process: subprocess.Popen[str] | None = None + + def is_cancelled(self) -> bool: + return self._event.is_set() + + def cancel(self) -> None: + self._event.set() + with self._lock: + process = self._process + if process is not None: + _terminate_process(process) + + def attach(self, process: subprocess.Popen[str]) -> None: + with self._lock: + self._process = process + cancelled = self._event.is_set() + if cancelled: + _terminate_process(process) + + def detach(self, process: subprocess.Popen[str]) -> None: + with self._lock: + if self._process is process: + self._process = None + + +TEMPLATE_FILES: dict[TemplateName, str] = { + "default": "swebench_template.yaml", + "qwen_tools": "swebench_qwen_tools_template.yaml", +} + +_LOG_TAIL_MAX_BYTES = 64 * 1024 +_LOG_TAIL_MAX_LINES = 50 + + +def _normalize_endpoint_base(endpoint: str) -> str: + base = endpoint.rstrip("/") + if base.endswith("/v1"): + base = base[:-3] + parsed = urlparse(base) + if parsed.hostname == "localhost": + netloc = "127.0.0.1" + if parsed.port is not None: + netloc = f"{netloc}:{parsed.port}" + base = urlunparse(parsed._replace(netloc=netloc)) + return base + + +def _exact_instance_filter(instance_ids: list[str]) -> str: + return ( + "^(?:" + "|".join(re.escape(instance_id) for instance_id in instance_ids) + ")$" + ) + + +def _terminate_process(process: subprocess.Popen[str]) -> None: + if process.poll() is not None: + return + try: + if os.name == "nt": + process.terminate() + else: + os.killpg(process.pid, signal.SIGTERM) + process.wait(timeout=10) + except ProcessLookupError: + return + except subprocess.TimeoutExpired: + if os.name == "nt": + process.kill() + else: + os.killpg(process.pid, signal.SIGKILL) + process.wait(timeout=10) + + +def _run_subprocess( + cmd: list[str], + log_path: Path, + *, + cwd: Path, + timeout_s: int, + env: dict[str, str] | None = None, + cancel_token: CancellationToken | None = None, +) -> None: + if cancel_token is not None and cancel_token.is_cancelled(): + raise RunCancelled(f"subprocess cancelled before start: {cmd}") + process: subprocess.Popen[str] | None = None + try: + with log_path.open("w", encoding="utf-8") as log_file: + process = subprocess.Popen( + cmd, + stdin=subprocess.DEVNULL, + stdout=log_file, + stderr=subprocess.STDOUT, + text=True, + encoding="utf-8", + errors="replace", + cwd=str(cwd), + env=env, + start_new_session=os.name != "nt", + ) + if cancel_token is not None: + cancel_token.attach(process) + deadline = time.monotonic() + timeout_s + while True: + if cancel_token is not None and cancel_token.is_cancelled(): + _terminate_process(process) + process.communicate() + raise RunCancelled(f"subprocess cancelled: {cmd}") + remaining = deadline - time.monotonic() + if remaining <= 0: + _terminate_process(process) + process.communicate() + raise RunnerError(f"subprocess timed out after {timeout_s}s: {cmd}") + try: + process.communicate(timeout=min(0.5, remaining)) + if cancel_token is not None and cancel_token.is_cancelled(): + raise RunCancelled(f"subprocess cancelled: {cmd}") + break + except subprocess.TimeoutExpired: + continue + finally: + if process is not None and cancel_token is not None: + cancel_token.detach(process) + + if process.returncode != 0: + with log_path.open("rb") as log_file: + log_file.seek(0, os.SEEK_END) + size = log_file.tell() + log_file.seek(max(0, size - _LOG_TAIL_MAX_BYTES)) + tail_bytes = log_file.read() + tail = "\n".join( + tail_bytes.decode("utf-8", errors="replace").splitlines()[ + -_LOG_TAIL_MAX_LINES: + ] + ) + raise RunnerError( + f"subprocess exited with code {process.returncode}: {cmd}\n{tail}" + ) + + +class SwebenchRunner: + def __init__( + self, + *, + project_root: Path, + subprocess_timeout_s: int, + ): + self.project_root = project_root.resolve() + self.subprocess_timeout_s = subprocess_timeout_s + + def run( + self, + request: RunRequest, + run_dir: Path, + cancel_token: CancellationToken | None = None, + ) -> dict[str, Any]: + run_dir.mkdir(parents=True, exist_ok=True) + secret_values = ( + {request.endpoint_api_key} if request.endpoint_api_key else set() + ) + (run_dir / "request.json").write_bytes( + msgspec.json.encode( + redact_secrets(request.model_dump(), secret_values=secret_values) + ) + ) + + output_dir = run_dir / "swe_bench_output" + if output_dir.exists(): + shutil.rmtree(output_dir) + output_dir.mkdir(parents=True) + + with tempfile.TemporaryDirectory(prefix="swebench_config_") as config_tmp: + patched_config = self._patch_config( + Path(config_tmp), + request, + ) + self._run_agent(request, patched_config, output_dir, run_dir, cancel_token) + + preds_path = output_dir / "preds.json" + if not preds_path.exists(): + raise RunnerError("mini-extra did not produce preds.json") + self._validate_prediction_ids(request, preds_path) + shutil.copy2(preds_path, run_dir / "preds.json") + + result_path = self._run_eval( + request, preds_path, output_dir, run_dir, cancel_token + ) + shutil.copy2(result_path, run_dir / "swe_bench_results.json") + return msgspec.json.decode(result_path.read_bytes(), type=dict) + + def _load_template(self, request: RunRequest) -> dict[str, Any]: + template_path = self._template_dir / TEMPLATE_FILES[request.template] + with template_path.open() as f: + loaded = yaml.safe_load(f) + if not isinstance(loaded, dict): + raise RunnerError("swebench template must be a YAML mapping") + model_cfg = loaded.get("model") + if not isinstance(model_cfg, dict): + raise RunnerError("swebench template must define model") + if not isinstance(model_cfg.get("model_kwargs"), dict): + raise RunnerError("swebench template must define model.model_kwargs") + return loaded + + @property + def _template_dir(self) -> Path: + return Path(__file__).resolve().parent / "templates" + + def _patch_config(self, config_dir: Path, request: RunRequest) -> Path: + cfg = self._load_template(request) + model_cfg = cfg["model"] + model_kwargs = model_cfg["model_kwargs"] + + model_cfg["model_name"] = request.model_name + if request.endpoint_urls: + base = _normalize_endpoint_base(str(request.endpoint_urls[0])) + model_kwargs["api_base"] = base + "/v1" + else: + base = "" + model_kwargs["api_base"] = "" + + if request.endpoint_api_key: + model_kwargs["api_key"] = request.endpoint_api_key + elif urlparse(base).hostname in {"localhost", "127.0.0.1", "::1"}: + model_kwargs["api_key"] = "EMPTY" + else: + model_kwargs.pop("api_key", None) + + for field in ( + "temperature", + "top_p", + "top_k", + "repetition_penalty", + "presence_penalty", + "frequency_penalty", + ): + val = request.generation_params.get(field) + if val is not None: + model_kwargs[field] = val + else: + model_kwargs.pop(field, None) + + if ( + max_new_tokens := request.generation_params.get("max_new_tokens") + ) is not None: + model_kwargs["max_tokens"] = max_new_tokens + else: + model_kwargs.pop("max_tokens", None) + + if ( + chat_tmpl := request.generation_params.get("chat_template_kwargs") + ) is not None: + model_kwargs["chat_template_kwargs"] = chat_tmpl + else: + model_kwargs.pop("chat_template_kwargs", None) + + config_dir.mkdir(parents=True, exist_ok=True) + patched_path = config_dir / "swebench_patched.yaml" + with patched_path.open("w") as f: + yaml.safe_dump(cfg, f, default_flow_style=False, sort_keys=False) + return patched_path + + def _run_agent( + self, + request: RunRequest, + patched_config: Path, + output_dir: Path, + run_dir: Path, + cancel_token: CancellationToken | None = None, + ) -> None: + instance_filter = _exact_instance_filter(request.evaluated_instance_ids) + cmd = [ + "mini-extra", + "swebench", + "--model", + request.model_name, + "--config", + str(patched_config), + "--subset", + request.subset, + "--split", + request.split, + "--filter", + instance_filter, + "--workers", + str(request.workers), + "--output", + str(output_dir), + ] + if request.enable_swebench_toolcall_patch: + with tempfile.TemporaryDirectory(prefix="minisweagent_overlay_") as tmp: + env = self._agent_env(request, Path(tmp)) + _run_subprocess( + cmd, + run_dir / "swe_bench_agent.log", + cwd=output_dir, + timeout_s=self.subprocess_timeout_s, + env=env, + cancel_token=cancel_token, + ) + return + _run_subprocess( + cmd, + run_dir / "swe_bench_agent.log", + cwd=output_dir, + timeout_s=self.subprocess_timeout_s, + env=self._base_env(request), + cancel_token=cancel_token, + ) + + def _base_env(self, request: RunRequest) -> dict[str, str]: + env = dict(os.environ) + no_proxy = {"127.0.0.1", "localhost"} + for endpoint in request.endpoint_urls: + host = urlparse(str(endpoint)).hostname + if host: + no_proxy.add(host) + existing = env.get("NO_PROXY") or env.get("no_proxy") + if existing: + no_proxy.update( + part.strip() for part in existing.split(",") if part.strip() + ) + no_proxy_value = ",".join(sorted(no_proxy)) + env["NO_PROXY"] = no_proxy_value + env["no_proxy"] = no_proxy_value + return env + + def _agent_env(self, request: RunRequest, overlay_root: Path) -> dict[str, str]: + env = self._base_env(request) + overlay = self._create_toolcall_patch_overlay(overlay_root, self._template_dir) + pythonpath = [str(overlay)] + if existing := env.get("PYTHONPATH"): + pythonpath.append(existing) + env["PYTHONPATH"] = os.pathsep.join(pythonpath) + return env + + def _create_toolcall_patch_overlay( + self, overlay_root: Path, replacement_root: Path + ) -> Path: + site_packages = self._resolve_minisweagent_site_packages() + package_src = site_packages / "minisweagent" + if not package_src.is_dir(): + raise RunnerError( + f"minisweagent package directory not found: {package_src}" + ) + package_dest = overlay_root / "minisweagent" + shutil.copytree(package_src, package_dest) + replacements = { + "actions_toolcall.py": "minisweagent/models/utils/actions_toolcall.py", + "litellm_model.py": "minisweagent/models/litellm_model.py", + } + for src_name, rel_dest in replacements.items(): + src = replacement_root / src_name + if not src.exists(): + raise RunnerError( + "enable_swebench_toolcall_patch requested, but replacement " + f"file is missing on the service host: {src}" + ) + shutil.copy2(src, overlay_root / rel_dest) + return overlay_root + + def _validate_prediction_ids(self, request: RunRequest, preds_path: Path) -> None: + try: + preds = msgspec.json.decode(preds_path.read_bytes(), type=dict) + except msgspec.DecodeError as exc: + raise RunnerError("mini-extra produced invalid preds.json") from exc + expected = set(request.evaluated_instance_ids) + actual = {str(instance_id) for instance_id in preds} + unexpected = sorted(actual - expected) + if unexpected: + raise RunnerError( + "mini-extra produced predictions for unexpected SWE-bench " + f"instances: {', '.join(unexpected[:10])}" + ) + + def _resolve_minisweagent_site_packages(self) -> Path: + result = subprocess.run( + [ + sys.executable, + "-c", + "import minisweagent.models.utils.actions_toolcall as m; print(m.__file__)", + ], + check=False, + capture_output=True, + text=True, + timeout=30, + ) + if result.returncode != 0: + raise RunnerError("could not locate minisweagent: " + result.stderr.strip()) + last_line = next( + (line for line in reversed(result.stdout.splitlines()) if line.strip()), + "", + ) + if not last_line: + raise RunnerError("could not locate minisweagent: empty output") + actions_toolcall = Path(last_line.strip()) + try: + site_packages = actions_toolcall.parents[3] + except IndexError as exc: + raise RunnerError( + f"could not resolve site-packages from {actions_toolcall}" + ) from exc + if not site_packages.is_dir(): + raise RunnerError(f"resolved site-packages does not exist: {site_packages}") + return site_packages + + def _run_eval( + self, + request: RunRequest, + preds_path: Path, + output_dir: Path, + run_dir: Path, + cancel_token: CancellationToken | None = None, + ) -> Path: + run_id = f"endpoints_{uuid.uuid4().hex[:8]}" + (run_dir / "swe_bench_eval_run_id.txt").write_text(run_id) + dataset_name = { + "verified": "princeton-nlp/SWE-bench_Verified", + "lite": "princeton-nlp/SWE-bench_Lite", + }.get(request.subset) + if dataset_name is None: + raise RunnerError(f"unknown SWE-bench subset: {request.subset}") + cmd = [ + sys.executable, + "-m", + "swebench.harness.run_evaluation", + "--dataset_name", + dataset_name, + "--split", + request.split, + "--predictions_path", + str(preds_path), + "--max_workers", + str(request.max_eval_workers), + "--run_id", + run_id, + "--instance_ids", + *request.evaluated_instance_ids, + ] + _run_subprocess( + cmd, + run_dir / "swe_bench_eval.log", + cwd=output_dir, + timeout_s=self.subprocess_timeout_s, + cancel_token=cancel_token, + ) + safe_model = request.model_name.replace("/", "__") + result_path = output_dir / f"{safe_model}.{run_id}.json" + if result_path.exists(): + return result_path + candidates = sorted(output_dir.rglob(f"*{run_id}*.json")) + if not candidates: + raise RunnerError(f"SWE-bench result file not found for run_id={run_id}") + return candidates[0] diff --git a/src/inference_endpoint/evaluation/swebench_service/swebench_service/schemas.py b/src/inference_endpoint/evaluation/swebench_service/swebench_service/schemas.py new file mode 100644 index 000000000..bbdd35421 --- /dev/null +++ b/src/inference_endpoint/evaluation/swebench_service/swebench_service/schemas.py @@ -0,0 +1,69 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +from typing import Any, Literal + +from pydantic import BaseModel, ConfigDict, Field + +RunState = Literal["queued", "running", "succeeded", "failed", "cancelled"] +RunProgressPhase = Literal[ + "queued", + "agent", + "eval", + "succeeded", + "failed", + "cancelled", +] +TemplateName = Literal["default", "qwen_tools"] + + +class RunRequest(BaseModel): + model_config = ConfigDict(extra="forbid") + + model_name: str = Field(min_length=1) + endpoint_urls: list[str] = Field(min_length=1, max_length=1) + endpoint_api_key: str | None = None + generation_params: dict[str, Any] = Field(default_factory=dict) + subset: str = "verified" + split: str = "test" + num_instances: int = Field(ge=1) + workers: int = Field(ge=1) + max_eval_workers: int = Field(ge=1) + evaluated_instance_ids: list[str] = Field(min_length=1) + enable_swebench_toolcall_patch: bool = False + template: TemplateName = "default" + + +class ArtifactInfo(BaseModel): + name: str + url: str + + +class RunStatus(BaseModel): + run_id: str + status: RunState + created_at: float + updated_at: float + phase: RunProgressPhase | None = None + agent_total: int | None = None + agent_completed: int | None = None + eval_total: int | None = None + eval_completed: int | None = None + message: str | None = None + error: str | None = None + result: dict[str, Any] | None = None + artifacts: list[ArtifactInfo] = Field(default_factory=list) diff --git a/src/inference_endpoint/evaluation/swebench_service/swebench_service/server.py b/src/inference_endpoint/evaluation/swebench_service/swebench_service/server.py new file mode 100644 index 000000000..101661e1a --- /dev/null +++ b/src/inference_endpoint/evaluation/swebench_service/swebench_service/server.py @@ -0,0 +1,444 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import asyncio +import inspect +import shutil +import time +import uuid +from pathlib import Path +from typing import Any + +import msgspec.json +import yaml +from aiohttp import web +from pydantic import ValidationError + +from . import API_VERSION, CAPABILITIES +from .artifacts import redact_secrets, redact_text, resolve_artifact +from .config import ServiceConfig +from .runner import CancellationToken, RunCancelled, SwebenchRunner +from .schemas import ArtifactInfo, RunRequest, RunStatus + + +class RunManager: + def __init__(self, *, config: ServiceConfig, runner: Any): + self.config = config + self.runner = runner + self.runs: dict[str, RunStatus] = {} + self._requests: dict[str, RunRequest] = {} + self._tasks: dict[str, asyncio.Task] = {} + self._cancel_tokens: dict[str, CancellationToken] = {} + self._secret_values: dict[str, set[str]] = {} + self._semaphore = asyncio.Semaphore(config.max_concurrent_runs) + self._submit_lock = asyncio.Lock() + + def active_count(self) -> int: + return sum( + 1 for run in self.runs.values() if run.status in {"queued", "running"} + ) + + async def submit(self, request: RunRequest) -> RunStatus: + async with self._submit_lock: + if self.active_count() >= self.config.max_concurrent_runs: + raise web.HTTPTooManyRequests(text="too many active SWE-bench runs") + run_id = uuid.uuid4().hex + now = time.time() + status = RunStatus( + run_id=run_id, + status="queued", + created_at=now, + updated_at=now, + phase="queued", + agent_total=len(request.evaluated_instance_ids), + agent_completed=0, + eval_total=0, + eval_completed=0, + message="queued", + ) + self.runs[run_id] = status + self._requests[run_id] = request + self._cancel_tokens[run_id] = CancellationToken() + self._secret_values[run_id] = self._secrets_for_request(request) + run_dir = self.run_dir(run_id) + run_dir.mkdir(parents=True, exist_ok=True) + (run_dir / "request.json").write_bytes( + msgspec.json.encode( + redact_secrets( + request.model_dump(mode="json"), + secret_values=self._secret_values[run_id], + ) + ) + ) + self._write_status(status) + self._tasks[run_id] = asyncio.create_task( + self._execute(run_id, request, run_dir) + ) + return status + + async def _execute(self, run_id: str, request: RunRequest, run_dir: Path) -> None: + token = self._cancel_tokens[run_id] + try: + async with self._semaphore: + if token.is_cancelled(): + self._update(run_id, status="cancelled", phase="cancelled") + return + self._update(run_id, status="running", phase="agent", message="agent") + try: + result = await asyncio.to_thread( + self._run_runner, request, run_dir, token + ) + except RunCancelled: + self._update(run_id, status="cancelled", phase="cancelled") + return + except Exception as exc: + if token.is_cancelled(): + self._update(run_id, status="cancelled", phase="cancelled") + return + self._update( + run_id, + status="failed", + phase="failed", + error=redact_text( + str(exc), self._secret_values.get(run_id, set()) + ), + ) + return + if token.is_cancelled(): + self._update(run_id, status="cancelled", phase="cancelled") + return + self._refresh_progress(run_id) + final_progress = self._terminal_progress(run_id, "succeeded") + artifacts = [ + ArtifactInfo(name=name, url=f"/v1/runs/{run_id}/artifacts/{name}") + for name in ( + "preds.json", + "swe_bench_agent.log", + "swe_bench_eval.log", + "swe_bench_results.json", + "status.json", + ) + if (run_dir / name).exists() + ] + self._update( + run_id, + status="succeeded", + result=result, + artifacts=artifacts, + **final_progress, + ) + finally: + self._tasks.pop(run_id, None) + self._prune_completed_runs() + + def _run_runner( + self, request: RunRequest, run_dir: Path, token: CancellationToken + ) -> dict[str, Any]: + try: + signature = inspect.signature(self.runner.run) + except (TypeError, ValueError): + return self.runner.run(request, run_dir) + if "cancel_token" not in signature.parameters: + return self.runner.run(request, run_dir) + return self.runner.run(request, run_dir, cancel_token=token) + + def cancel(self, run_id: str) -> RunStatus: + status = self.get(run_id) + if status.status not in {"queued", "running"}: + return status + token = self._cancel_tokens.get(run_id) + if token is not None: + token.cancel() + self._update( + run_id, status="cancelled", **self._terminal_progress(run_id, "cancelled") + ) + return self.runs[run_id] + + async def cancel_all_active(self) -> None: + active = [ + run.run_id + for run in self.runs.values() + if run.status in {"queued", "running"} + ] + for run_id in active: + self.cancel(run_id) + tasks = [self._tasks[run_id] for run_id in active if run_id in self._tasks] + if tasks: + await asyncio.gather(*tasks, return_exceptions=True) + + def _secrets_for_request(self, request: RunRequest) -> set[str]: + secrets: set[str] = set() + if request.endpoint_api_key: + secrets.add(request.endpoint_api_key) + if self.config.auth_token: + secrets.add(self.config.auth_token) + return secrets + + def _update(self, run_id: str, **updates: Any) -> None: + current = self.runs[run_id] + data = current.model_dump() + data.update(updates) + data["updated_at"] = time.time() + updated = RunStatus.model_validate(data) + self.runs[run_id] = updated + self._write_status(updated) + + def _write_status(self, status: RunStatus) -> None: + run_dir = self.run_dir(status.run_id) + run_dir.mkdir(parents=True, exist_ok=True) + secret_values = self._secret_values.get(status.run_id, set()) + (run_dir / "status.json").write_bytes( + msgspec.json.encode( + redact_secrets( + status.model_dump(mode="json"), secret_values=secret_values + ) + ) + ) + + def get(self, run_id: str) -> RunStatus: + try: + self._refresh_progress(run_id) + return self.runs[run_id] + except KeyError as exc: + raise web.HTTPNotFound(text="unknown run_id") from exc + + def _refresh_progress(self, run_id: str) -> None: + status = self.runs[run_id] + if status.status in {"failed", "cancelled"}: + progress = self._terminal_progress(run_id, status.status) + elif status.status == "succeeded": + progress = self._terminal_progress(run_id, "succeeded") + else: + progress = self._derived_progress(run_id) + + current = status.model_dump() + changed = any(current.get(key) != value for key, value in progress.items()) + if not changed: + return + current.update(progress) + current["updated_at"] = time.time() + updated = RunStatus.model_validate(current) + self.runs[run_id] = updated + self._write_status(updated) + + def _derived_progress(self, run_id: str) -> dict[str, Any]: + request = self._requests.get(run_id) + run_dir = self.run_dir(run_id) + output_dir = run_dir / "swe_bench_output" + agent_total = len(request.evaluated_instance_ids) if request else 0 + agent_completed = min(self._count_agent_completed(output_dir), agent_total) + + eval_run_id_path = run_dir / "swe_bench_eval_run_id.txt" + eval_total = self._count_predictions(output_dir / "preds.json") or agent_total + eval_completed = 0 + if eval_run_id_path.exists(): + phase = "eval" + eval_run_id = eval_run_id_path.read_text().strip() + eval_completed = min( + self._count_eval_completed(output_dir, eval_run_id), + eval_total, + ) + message = "eval" + else: + phase = "queued" if self.runs[run_id].status == "queued" else "agent" + eval_total = 0 + message = phase + + return { + "phase": phase, + "agent_total": agent_total, + "agent_completed": agent_completed, + "eval_total": eval_total, + "eval_completed": eval_completed, + "message": message, + } + + def _terminal_progress(self, run_id: str, phase: str) -> dict[str, Any]: + progress = self._derived_progress(run_id) + if phase == "succeeded": + agent_total = progress["agent_total"] or 0 + eval_total = progress["eval_total"] or self._count_predictions( + self.run_dir(run_id) / "swe_bench_output" / "preds.json" + ) + if not eval_total: + eval_total = agent_total + progress["agent_completed"] = agent_total + progress["eval_total"] = eval_total + progress["eval_completed"] = eval_total + progress["phase"] = phase + progress["message"] = phase + return progress + + @staticmethod + def _count_agent_completed(output_dir: Path) -> int: + exit_statuses = sorted( + output_dir.glob("exit_statuses_*.yaml"), + key=lambda path: path.stat().st_mtime, + ) + if exit_statuses: + try: + loaded = yaml.safe_load(exit_statuses[-1].read_text()) or {} + except (OSError, yaml.YAMLError): + loaded = {} + instances_by_status = loaded.get("instances_by_exit_status") + if isinstance(instances_by_status, dict): + completed: set[str] = set() + for instance_ids in instances_by_status.values(): + if isinstance(instance_ids, list): + completed.update( + str(instance_id) for instance_id in instance_ids + ) + return len(completed) + + traj_ids = { + path.name.removesuffix(".traj.json") + for path in output_dir.glob("*/*.traj.json") + } + if traj_ids: + return len(traj_ids) + return RunManager._count_predictions(output_dir / "preds.json") + + @staticmethod + def _count_predictions(preds_path: Path) -> int: + if not preds_path.exists(): + return 0 + try: + loaded = msgspec.json.decode(preds_path.read_bytes(), type=dict) + except (OSError, msgspec.DecodeError): + return 0 + return len(loaded) + + @staticmethod + def _count_eval_completed(output_dir: Path, eval_run_id: str) -> int: + if not eval_run_id: + return 0 + run_root = output_dir / "logs" / "run_evaluation" / eval_run_id + if not run_root.exists(): + return 0 + return sum(1 for _ in run_root.rglob("report.json")) + + def run_dir(self, run_id: str) -> Path: + return self.config.artifact_root / run_id + + def _prune_completed_runs(self) -> None: + limit = self.config.max_stored_runs + if limit <= 0: + return + completed = [ + run for run in self.runs.values() if run.status not in {"queued", "running"} + ] + overflow = len(completed) - limit + if overflow <= 0: + return + for run in sorted(completed, key=lambda item: item.updated_at)[:overflow]: + self.runs.pop(run.run_id, None) + self._requests.pop(run.run_id, None) + self._cancel_tokens.pop(run.run_id, None) + self._secret_values.pop(run.run_id, None) + shutil.rmtree(self.run_dir(run.run_id), ignore_errors=True) + + +MANAGER_KEY = web.AppKey("manager", RunManager) + + +def create_app(config: ServiceConfig, runner: Any | None = None) -> web.Application: + config = ServiceConfig( + host=config.host, + port=config.port, + artifact_root=config.artifact_root.expanduser().resolve(), + max_concurrent_runs=config.max_concurrent_runs, + subprocess_timeout_s=config.subprocess_timeout_s, + auth_token=config.auth_token, + max_stored_runs=config.max_stored_runs, + ) + config.artifact_root.mkdir(parents=True, exist_ok=True) + runner = runner or SwebenchRunner( + project_root=Path(__file__).resolve().parents[1], + subprocess_timeout_s=config.subprocess_timeout_s, + ) + manager = RunManager(config=config, runner=runner) + + @web.middleware + async def auth_middleware(request: web.Request, handler: Any) -> web.StreamResponse: + if config.auth_token and request.headers.get("Authorization") != ( + f"Bearer {config.auth_token}" + ): + raise web.HTTPUnauthorized(text="unauthorized") + return await handler(request) + + app = web.Application(middlewares=[auth_middleware]) + app[MANAGER_KEY] = manager + + async def shutdown_active_runs(app: web.Application) -> None: + await app[MANAGER_KEY].cancel_all_active() + + app.on_shutdown.append(shutdown_active_runs) + + async def health(request: web.Request) -> web.Response: + return web.json_response( + {"api_version": API_VERSION, "capabilities": CAPABILITIES, "status": "ok"} + ) + + async def post_run(request: web.Request) -> web.Response: + try: + data = await request.json() + run_request = RunRequest.model_validate(data) + except (ValidationError, ValueError) as exc: + raise web.HTTPBadRequest(text="invalid run request") from exc + status = await manager.submit(run_request) + return web.json_response( + redact_secrets( + status.model_dump(mode="json"), + secret_values=manager._secret_values.get(status.run_id, set()), + ), + status=202, + ) + + async def get_run(request: web.Request) -> web.Response: + status = manager.get(request.match_info["run_id"]) + return web.json_response( + redact_secrets( + status.model_dump(mode="json"), + secret_values=manager._secret_values.get(status.run_id, set()), + ) + ) + + async def cancel_run(request: web.Request) -> web.Response: + status = manager.cancel(request.match_info["run_id"]) + return web.json_response( + redact_secrets( + status.model_dump(mode="json"), + secret_values=manager._secret_values.get(status.run_id, set()), + ) + ) + + async def get_artifact(request: web.Request) -> web.FileResponse: + run_id = request.match_info["run_id"] + name = request.match_info["name"] + manager.get(run_id) + try: + path = resolve_artifact(manager.run_dir(run_id), name) + except FileNotFoundError as exc: + raise web.HTTPNotFound(text="artifact not found") from exc + return web.FileResponse(path) + + app.router.add_get("/health", health) + app.router.add_post("/v1/runs", post_run) + app.router.add_get("/v1/runs/{run_id}", get_run) + app.router.add_post("/v1/runs/{run_id}/cancel", cancel_run) + app.router.add_get("/v1/runs/{run_id}/artifacts/{name}", get_artifact) + return app diff --git a/src/inference_endpoint/evaluation/swebench_service/swebench_service/templates/actions_toolcall.py b/src/inference_endpoint/evaluation/swebench_service/swebench_service/templates/actions_toolcall.py new file mode 100644 index 000000000..cc6ed6cdf --- /dev/null +++ b/src/inference_endpoint/evaluation/swebench_service/swebench_service/templates/actions_toolcall.py @@ -0,0 +1,283 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Parse actions & format observations with toolcalls""" + +import base64 +import json +import shlex +import time + +from jinja2 import StrictUndefined, Template +from minisweagent.exceptions import FormatError +from minisweagent.models.utils.openai_multimodal import expand_multimodal_content + +BASH_TOOL = { + "type": "function", + "function": { + "name": "bash", + "description": "Execute a bash command", + "parameters": { + "type": "object", + "properties": { + "command": { + "type": "string", + "description": "The bash command to execute", + } + }, + "required": ["command"], + }, + }, +} + +FINISH_TOOL = { + "type": "function", + "function": { + "name": "finish", + "description": ( + "Submit your solution when complete. The patch is automatically extracted " + "from your git changes — do NOT create patch.txt manually." + ), + "parameters": { + "type": "object", + "properties": { + "files_modified": { + "type": "array", + "items": {"type": "string"}, + "description": ( + "Source file paths you modified, relative to /testbed " + "(e.g. ['django/db/models/query.py'])" + ), + } + }, + "required": ["files_modified"], + }, + }, +} + +STR_REPLACE_EDITOR_TOOL = { + "type": "function", + "function": { + "name": "str_replace_editor", + "description": ( + "View or edit files precisely. Use 'view' to read a file with line numbers, " + "'str_replace' to replace an exact string (must match exactly once), " + "'create' to write a new file." + ), + "parameters": { + "type": "object", + "properties": { + "command": { + "type": "string", + "enum": ["view", "str_replace", "create"], + }, + "path": { + "type": "string", + "description": "Absolute file path", + }, + "old_str": { + "type": "string", + "description": "Exact string to replace (str_replace only)", + }, + "new_str": { + "type": "string", + "description": "Replacement string (str_replace) or file content (create)", + }, + "view_range": { + "type": "array", + "items": {"type": "integer"}, + "description": "Optional [start_line, end_line] for view (1-indexed)", + }, + }, + "required": ["command", "path"], + }, + }, +} + +TOOL_SCHEMAS = [BASH_TOOL, FINISH_TOOL, STR_REPLACE_EDITOR_TOOL] + + +def parse_toolcall_actions( + tool_calls: list, *, format_error_template: str +) -> list[dict]: + """Parse tool calls from the response. Raises FormatError if unknown tool or invalid args.""" + if not tool_calls: + raise FormatError( + { + "role": "user", + "content": Template( + format_error_template, undefined=StrictUndefined + ).render( + error="No tool calls found in the response. Every response MUST include at least one tool call.", + actions=[], + ), + "extra": {"interrupt_type": "FormatError"}, + } + ) + actions = [] + errors = [] + for tool_call in tool_calls: + args = {} + try: + args = json.loads(tool_call.function.arguments) + except Exception as e: + errors.append(f"Error parsing tool call arguments: {e}.") + continue + name = tool_call.function.name + tool_call_id = tool_call.id + + if name == "bash": + if not isinstance(args, dict) or "command" not in args: + errors.append("Missing 'command' argument in bash tool call.") + continue + actions.append({"command": args["command"], "tool_call_id": tool_call_id}) + + elif name == "finish": + files = args.get("files_modified", []) + if not isinstance(files, list) or not files: + errors.append( + "finish: files_modified must be a non-empty list of file paths." + ) + continue + + def _to_rel(p: str) -> str: + p = p.lstrip("/") + if p.startswith("testbed/"): + p = p[len("testbed/") :] + return p + + files_str = " ".join(shlex.quote(_to_rel(f)) for f in files) + actions.append( + { + "command": ( + "echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT && " + f"git -C /testbed add -N -- {files_str} && " + f"git -C /testbed diff HEAD -- {files_str}" + ), + "tool_call_id": tool_call_id, + } + ) + + elif name == "str_replace_editor": + command = args.get("command") + path = args.get("path", "") + if command == "view": + view_range = args.get("view_range") + if view_range is not None: + if not isinstance(view_range, list | tuple) or len(view_range) != 2: + errors.append( + "str_replace_editor view_range must be [start, end]." + ) + continue + try: + start, end = int(view_range[0]), int(view_range[1]) + except (TypeError, ValueError): + errors.append( + "str_replace_editor view_range values must be integers." + ) + continue + bash_cmd = ( + f"awk 'NR>={start} && NR<={end} " + f'{{printf "%6d\\t%s\\n", NR, $0}}\' {shlex.quote(path)}' + ) + else: + bash_cmd = f"cat -n {shlex.quote(path)}" + actions.append({"command": bash_cmd, "tool_call_id": tool_call_id}) + elif command == "str_replace": + old_str = args.get("old_str", "") + new_str = args.get("new_str", "") + path_b64 = base64.b64encode(path.encode()).decode() + old_b64 = base64.b64encode(old_str.encode()).decode() + new_b64 = base64.b64encode(new_str.encode()).decode() + bash_cmd = ( + f"python3 -c 'import base64,sys;" + f'path=base64.b64decode("{path_b64}").decode();' + f'old=base64.b64decode("{old_b64}").decode();' + f'new=base64.b64decode("{new_b64}").decode();' + f"c=open(path).read();n=c.count(old);" + f'sys.exit("Expected exactly 1 match, found "+str(n)) if n!=1 else open(path,"w").write(c.replace(old,new,1))\'' + ) + actions.append({"command": bash_cmd, "tool_call_id": tool_call_id}) + elif command == "create": + content = args.get("new_str", "") + path_b64 = base64.b64encode(path.encode()).decode() + content_b64 = base64.b64encode(content.encode()).decode() + bash_cmd = ( + f"python3 -c 'import base64,os;" + f'path=base64.b64decode("{path_b64}").decode();' + f'os.makedirs(os.path.dirname(path) or ".",exist_ok=True);' + f'open(path,"w").write(base64.b64decode("{content_b64}").decode())\'' + ) + actions.append({"command": bash_cmd, "tool_call_id": tool_call_id}) + else: + errors.append(f"str_replace_editor: unknown command {command!r}.") + continue + + else: + errors.append(f"Unknown tool '{name}'.") + continue + + if errors: + raise FormatError( + { + "role": "user", + "content": Template( + format_error_template, undefined=StrictUndefined + ).render(actions=actions, error=" ".join(errors).strip()), + "extra": {"interrupt_type": "FormatError"}, + } + ) + return actions + + +def format_toolcall_observation_messages( + *, + actions: list[dict], + outputs: list[dict], + observation_template: str, + template_vars: dict | None = None, + multimodal_regex: str = "", +) -> list[dict]: + """Format execution outputs into tool result messages.""" + not_executed = { + "output": "", + "returncode": -1, + "exception_info": "action was not executed", + } + padded_outputs = outputs + [not_executed] * (len(actions) - len(outputs)) + results = [] + for action, output in zip(actions, padded_outputs, strict=False): + content = Template(observation_template, undefined=StrictUndefined).render( + output=output, **(template_vars or {}) + ) + msg = { + "content": content, + "extra": { + "raw_output": output.get("output", ""), + "returncode": output.get("returncode"), + "timestamp": time.time(), + "exception_info": output.get("exception_info"), + **output.get("extra", {}), + }, + } + if "tool_call_id" in action: + msg["tool_call_id"] = action["tool_call_id"] + msg["role"] = "tool" + else: + msg["role"] = "user" # human issued commands + if multimodal_regex: + msg = expand_multimodal_content(msg, pattern=multimodal_regex) + results.append(msg) + return results diff --git a/src/inference_endpoint/evaluation/swebench_service/swebench_service/templates/litellm_model.py b/src/inference_endpoint/evaluation/swebench_service/swebench_service/templates/litellm_model.py new file mode 100644 index 000000000..4d4e27a34 --- /dev/null +++ b/src/inference_endpoint/evaluation/swebench_service/swebench_service/templates/litellm_model.py @@ -0,0 +1,174 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import json +import logging +import os +import time +from collections.abc import Callable +from pathlib import Path +from typing import Any, Literal, cast + +import litellm +from minisweagent.models import GLOBAL_MODEL_STATS +from minisweagent.models.utils.actions_toolcall import ( + TOOL_SCHEMAS, + format_toolcall_observation_messages, + parse_toolcall_actions, +) +from minisweagent.models.utils.anthropic_utils import _reorder_anthropic_thinking_blocks +from minisweagent.models.utils.cache_control import set_cache_control +from minisweagent.models.utils.openai_multimodal import expand_multimodal_content +from minisweagent.models.utils.retry import retry +from pydantic import BaseModel + +logger = logging.getLogger("litellm_model") + + +class LitellmModelConfig(BaseModel): + model_name: str + """Model name. Highly recommended to include the provider in the model name, e.g., `anthropic/claude-sonnet-4-5-20250929`.""" + model_kwargs: dict[str, Any] = {} + """Additional arguments passed to the API.""" + litellm_model_registry: Path | str | None = os.getenv("LITELLM_MODEL_REGISTRY_PATH") + """Model registry for cost tracking and model metadata. See the local model guide (https://mini-swe-agent.com/latest/models/local_models/) for more details.""" + set_cache_control: Literal["default_end"] | None = None + """Set explicit cache control markers, for example for Anthropic models""" + cost_tracking: Literal["default", "ignore_errors"] = cast( + Literal["default", "ignore_errors"], + os.getenv("MSWEA_COST_TRACKING", "default"), + ) + """Cost tracking mode for this model. Can be "default" or "ignore_errors" (ignore errors/missing cost info)""" + format_error_template: str = "{{ error }}" + """Template used when the LM's output is not in the expected format.""" + observation_template: str = ( + "{% if output.exception_info %}{{output.exception_info}}\n{% endif %}" + "{{output.returncode}}\n\n{{output.output}}" + ) + """Template used to render the observation after executing an action.""" + multimodal_regex: str = "" + """Regex to extract multimodal content. Empty string disables multimodal processing.""" + + +class LitellmModel: + abort_exceptions: list[type[Exception]] = [ + litellm.exceptions.UnsupportedParamsError, + litellm.exceptions.NotFoundError, + litellm.exceptions.PermissionDeniedError, + litellm.exceptions.ContextWindowExceededError, + litellm.exceptions.AuthenticationError, + ] + + def __init__(self, *, config_class: Callable = LitellmModelConfig, **kwargs): + self.config = config_class(**kwargs) + if ( + self.config.litellm_model_registry + and Path(self.config.litellm_model_registry).is_file() + ): + litellm.utils.register_model( + json.loads(Path(self.config.litellm_model_registry).read_text()) + ) + + def _query(self, messages: list[dict[str, str]], **kwargs): + try: + return litellm.completion( + model=self.config.model_name, + messages=messages, + tools=TOOL_SCHEMAS, + **(self.config.model_kwargs | kwargs), + ) + except litellm.exceptions.AuthenticationError as e: + e.message += " You can permanently set your API key with `mini-extra config set KEY VALUE`." + raise e + + def _prepare_messages_for_api(self, messages: list[dict]) -> list[dict]: + prepared = [{k: v for k, v in msg.items() if k != "extra"} for msg in messages] + prepared = _reorder_anthropic_thinking_blocks(prepared) + return set_cache_control(prepared, mode=self.config.set_cache_control) + + def query(self, messages: list[dict[str, str]], **kwargs) -> dict: + for attempt in retry(logger=logger, abort_exceptions=self.abort_exceptions): + with attempt: + response = self._query( + self._prepare_messages_for_api(messages), **kwargs + ) + cost_output = self._calculate_cost(response) + GLOBAL_MODEL_STATS.add(cost_output["cost"]) + message = response.choices[0].message.model_dump() + message["extra"] = { + "actions": self._parse_actions(response), + "response": response.model_dump(), + **cost_output, + "timestamp": time.time(), + } + return message + + def _calculate_cost(self, response) -> dict[str, float]: + try: + cost = litellm.cost_calculator.completion_cost( + response, model=self.config.model_name + ) + if cost <= 0.0: + raise ValueError(f"Cost must be > 0.0, got {cost}") + except Exception as e: + cost = 0.0 + if self.config.cost_tracking != "ignore_errors": + msg = ( + f"Error calculating cost for model {self.config.model_name}: {e}, perhaps it's not registered? " + "You can ignore this issue from your config file with cost_tracking: 'ignore_errors' or " + "globally with export MSWEA_COST_TRACKING='ignore_errors'. " + "Alternatively check the 'Cost tracking' section in the documentation at " + "https://klieret.short.gy/mini-local-models. " + " Still stuck? Please open a GitHub issue at https://github.com/SWE-agent/mini-swe-agent/issues/new/choose!" + ) + logger.critical(msg) + raise RuntimeError(msg) from e + return {"cost": cost} + + def _parse_actions(self, response) -> list[dict]: + """Parse tool calls from the response. Raises FormatError if unknown tool.""" + tool_calls = response.choices[0].message.tool_calls or [] + return parse_toolcall_actions( + tool_calls, format_error_template=self.config.format_error_template + ) + + def format_message(self, **kwargs) -> dict: + return expand_multimodal_content(kwargs, pattern=self.config.multimodal_regex) + + def format_observation_messages( + self, message: dict, outputs: list[dict], template_vars: dict | None = None + ) -> list[dict]: + """Format execution outputs into tool result messages.""" + actions = message.get("extra", {}).get("actions", []) + return format_toolcall_observation_messages( + actions=actions, + outputs=outputs, + observation_template=self.config.observation_template, + template_vars=template_vars, + multimodal_regex=self.config.multimodal_regex, + ) + + def get_template_vars(self, **kwargs) -> dict[str, Any]: + return self.config.model_dump() + + def serialize(self) -> dict: + return { + "info": { + "config": { + "model": self.config.model_dump(mode="json"), + "model_type": f"{self.__class__.__module__}.{self.__class__.__name__}", + }, + } + } diff --git a/src/inference_endpoint/evaluation/swebench_service/swebench_service/templates/swebench_qwen_tools_template.yaml b/src/inference_endpoint/evaluation/swebench_service/swebench_service/templates/swebench_qwen_tools_template.yaml new file mode 100644 index 000000000..7b9d52f81 --- /dev/null +++ b/src/inference_endpoint/evaluation/swebench_service/swebench_service/templates/swebench_qwen_tools_template.yaml @@ -0,0 +1,172 @@ +agent: + system_template: | + You are a helpful assistant that can interact with a computer shell to solve programming tasks. + instance_template: | + + Consider the following PR description: + {{task}} + + + + # Task Instructions + + ## Overview + + You're a software engineer interacting continuously with a computer by submitting commands. + You'll be helping implement necessary changes to meet requirements in the PR description. + Your task is specifically to make changes to non-test files in the current directory in order to fix the issue described in the PR description in a way that is general and consistent with the codebase. + This is an interactive process where you will think and issue AT LEAST ONE command, see the result, then think and issue your next command(s). + + For each response: + + 1. Include a THOUGHT section explaining your reasoning and what you're trying to accomplish + 2. Provide one or more bash tool calls to execute + + ## Important Boundaries + + - MODIFY: Regular source code files in /testbed (this is the working directory for all your subsequent commands) + - DO NOT MODIFY: Tests, configuration files (pyproject.toml, setup.cfg, etc.) + + ## Recommended Workflow + + 1. Analyze the codebase by finding and reading relevant files + 2. Create a script to reproduce the issue + 3. Edit the source code: use `str_replace_editor` for precise edits (`str_replace` for targeted changes, `create` for new files), bash for everything else + 4. Verify your fix works by running your script again + 5. Test edge cases to ensure your fix is robust + + ## Command Execution Rules + + You are operating in an environment where + + 1. You issue at least one command + 2. The system executes the command(s) in a subshell + 3. You see the result(s) + 4. You write your next command(s) + + Each response should include: + + 1. **Reasoning text** where you explain your analysis and plan + 2. At least one tool call with your command + + **CRITICAL REQUIREMENTS:** + + - Your response SHOULD include reasoning text explaining what you're doing + - Your response MUST include AT LEAST ONE bash tool call. You can make MULTIPLE tool calls in a single response when the commands are independent (e.g., searching multiple files, reading different parts of the codebase). + - Directory or environment variable changes are not persistent. Every action is executed in a new subshell. + - However, you can prefix any action with `MY_ENV_VAR=MY_VALUE cd /path/to/working/dir && ...` or write/load environment variables from files + + Example of a CORRECT response: + + I need to understand the Builder-related code. Let me find relevant files and check the project structure. + + [Makes multiple bash tool calls: {"command": "ls -la"}, {"command": "find src -name '*.java' | grep -i builder"}, {"command": "cat README.md | head -50"}] + + + ## Environment Details + + - You have a full Linux shell environment + - Always use non-interactive flags (-y, -f) for commands + - Avoid interactive tools like vi, nano, or any that require user input + - You can use bash commands or invoke any tool that is available in the environment + - You can also create new tools or scripts to help you with the task + - If a tool isn't available, you can also install it + - Prefer `str_replace_editor` over sed/heredocs for file edits: + - `view` (with optional `view_range=[start, end]`) — read a file with line numbers + - `str_replace` — replace an exact string that matches exactly once + - `create` — write an entire new file + - Use bash for everything else (search, run tests, install packages, etc.) + + ## Submission + + When you've completed your work, call the `finish` tool with the list of source + files you modified: + + finish(files_modified=["path/to/file1.py", "path/to/file2.py"]) + + The patch is automatically extracted from your git changes. Do NOT create + patch.txt or use the `echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT` command. + + + Only list source files you modified to fix the issue — not test files, reproduction + scripts, helper tools, configuration files, or binary files. + + + + You CANNOT continue working (reading, editing, testing) in any way on this task + after calling finish. + + + step_limit: 250 + cost_limit: 3. + +environment: + cwd: "/testbed" + timeout: 300 + interpreter: ["bash", "-c"] + env: + PAGER: cat + MANPAGER: cat + LESS: -R + PIP_PROGRESS_BAR: "off" + TQDM_DISABLE: "1" + environment_class: docker + pull_timeout: 3600 + container_timeout: 10h + +model: + cost_tracking: "ignore_errors" + observation_template: | + {% if output.exception_info -%} + {{output.exception_info}} + {% endif -%} + {{output.returncode}} + {% if output.output | length < 10000 -%} + + {{ output.output -}} + + {%- else -%} + + The output of your last command was too long. + Please try a different command that produces less output. + If you're looking at a file you can try use head, tail or sed to view a smaller number of lines selectively. + If you're using grep or find and it produced too much output, you can use a more selective search pattern. + If you really need to see something from the full command's output, you can redirect output to a file and then search in that file. + + {%- set elided_chars = output.output | length - 10000 -%} + + {{ output.output[:5000] }} + + + {{ elided_chars }} characters elided + + + {{ output.output[-5000:] }} + + {%- endif -%} + format_error_template: | + Tool call error: + + + {{error}} + + + Here is general guidance on how to submit correct toolcalls: + + Every response MUST include at least one tool call. Available tools: + - bash: execute a shell command — {"command": "your_command_here"} + - str_replace_editor: view or edit files — {"command": "view"|"str_replace"|"create", "path": "...", ...} + - finish: submit your solution — {"files_modified": ["path/to/file.py", ...]} + + If you have completed your assignment, call the finish tool as described in the instructions. + # Patched at runtime by SWEBenchScorer from model_params and endpoint_config + model_name: "" + model_kwargs: + custom_llm_provider: "openai" + api_key: "test" + drop_params: true + parallel_tool_calls: true + api_base: "" + # Sampling parameters (temperature, top_p, top_k, etc.) are injected at + # runtime from the benchmark config's model_params block — absent here so + # the model's own defaults apply when not specified in model_params. diff --git a/src/inference_endpoint/evaluation/swebench_service/swebench_service/templates/swebench_template.yaml b/src/inference_endpoint/evaluation/swebench_service/swebench_service/templates/swebench_template.yaml new file mode 100644 index 000000000..082f3635a --- /dev/null +++ b/src/inference_endpoint/evaluation/swebench_service/swebench_service/templates/swebench_template.yaml @@ -0,0 +1,186 @@ +agent: + system_template: | + You are a helpful assistant that can interact with a computer shell to solve programming tasks. + instance_template: | + + Consider the following PR description: + {{task}} + + + + # Task Instructions + + ## Overview + + You're a software engineer interacting continuously with a computer by submitting commands. + You'll be helping implement necessary changes to meet requirements in the PR description. + Your task is specifically to make changes to non-test files in the current directory in order to fix the issue described in the PR description in a way that is general and consistent with the codebase. + This is an interactive process where you will think and issue AT LEAST ONE command, see the result, then think and issue your next command(s). + + For each response: + + 1. Include a THOUGHT section explaining your reasoning and what you're trying to accomplish + 2. Provide one or more bash tool calls to execute + + ## Important Boundaries + + - MODIFY: Regular source code files in /testbed (this is the working directory for all your subsequent commands) + - DO NOT MODIFY: Tests, configuration files (pyproject.toml, setup.cfg, etc.) + + ## Recommended Workflow + + 1. Analyze the codebase by finding and reading relevant files + 2. Create a script to reproduce the issue + 3. Edit the source code to resolve the issue + 4. Verify your fix works by running your script again + 5. Test edge cases to ensure your fix is robust + + ## Command Execution Rules + + You are operating in an environment where + + 1. You issue at least one command + 2. The system executes the command(s) in a subshell + 3. You see the result(s) + 4. You write your next command(s) + + Each response should include: + + 1. **Reasoning text** where you explain your analysis and plan + 2. At least one tool call with your command + + **CRITICAL REQUIREMENTS:** + + - Your response SHOULD include reasoning text explaining what you're doing + - Your response MUST include AT LEAST ONE bash tool call. You can make MULTIPLE tool calls in a single response when the commands are independent (e.g., searching multiple files, reading different parts of the codebase). + - Directory or environment variable changes are not persistent. Every action is executed in a new subshell. + - However, you can prefix any action with `MY_ENV_VAR=MY_VALUE cd /path/to/working/dir && ...` or write/load environment variables from files + + Example of a CORRECT response: + + I need to understand the Builder-related code. Let me find relevant files and check the project structure. + + [Makes multiple bash tool calls: {"command": "ls -la"}, {"command": "find src -name '*.java' | grep -i builder"}, {"command": "cat README.md | head -50"}] + + + ## Environment Details + + - You have a full Linux shell environment + - Always use non-interactive flags (-y, -f) for commands + - Avoid interactive tools like vi, nano, or any that require user input + - You can use bash commands or invoke any tool that is available in the environment + - You can also create new tools or scripts to help you with the task + - If a tool isn't available, you can also install it + + ## Submission + + When you've completed your work, you MUST submit your changes as a git patch. + Follow these steps IN ORDER, with SEPARATE commands: + + Step 1: Create the patch file + Run `git diff -- path/to/file1 path/to/file2 > patch.txt` listing only the source files you modified. + Do NOT commit your changes. + + + The patch must only contain changes to the specific source files you modified to fix the issue. + Do not submit file creations or changes to any of the following files: + + - test and reproduction files + - helper scripts, tests, or tools that you created + - installation, build, packaging, configuration, or setup scripts unless they are directly part of the issue you were fixing (you can assume that the environment is already set up for your client) + - binary or compiled files + + + Step 2: Verify your patch + Inspect patch.txt to confirm it only contains your intended changes and headers show `--- a/` and `+++ b/` paths. + + Step 3: Submit (EXACT command required) + You MUST use this EXACT command to submit: + + ```bash + echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT && cat patch.txt + ``` + + If the command fails (nonzero exit status), it will not submit. + + + - Creating/viewing the patch and submitting it MUST be separate commands (not combined with &&). + - If you modify patch.txt after verifying, you SHOULD verify again before submitting. + - You CANNOT continue working (reading, editing, testing) in any way on this task after submitting. + + + step_limit: 250 + cost_limit: 3. + +environment: + cwd: "/testbed" + timeout: 300 + interpreter: ["bash", "-c"] + env: + PAGER: cat + MANPAGER: cat + LESS: -R + PIP_PROGRESS_BAR: "off" + TQDM_DISABLE: "1" + environment_class: docker + pull_timeout: 3600 + container_timeout: 10h + +model: + cost_tracking: "ignore_errors" + observation_template: | + {% if output.exception_info -%} + {{output.exception_info}} + {% endif -%} + {{output.returncode}} + {% if output.output | length < 10000 -%} + + {{ output.output -}} + + {%- else -%} + + The output of your last command was too long. + Please try a different command that produces less output. + If you're looking at a file you can try use head, tail or sed to view a smaller number of lines selectively. + If you're using grep or find and it produced too much output, you can use a more selective search pattern. + If you really need to see something from the full command's output, you can redirect output to a file and then search in that file. + + {%- set elided_chars = output.output | length - 10000 -%} + + {{ output.output[:5000] }} + + + {{ elided_chars }} characters elided + + + {{ output.output[-5000:] }} + + {%- endif -%} + format_error_template: | + Tool call error: + + + {{error}} + + + Here is general guidance on how to submit correct toolcalls: + + Every response needs to use the 'bash' tool at least once to execute commands. + + Call the bash tool with your command as the argument: + - Tool: bash + - Arguments: {"command": "your_command_here"} + + If you have completed your assignment, please consult the first message about how to + submit your solution (you will not be able to continue working on this task after that). + # Patched at runtime by SWEBenchScorer from model_params and endpoint_config + model_name: "" + model_kwargs: + custom_llm_provider: "openai" + api_key: "test" + drop_params: true + parallel_tool_calls: true + api_base: "" + # Sampling parameters (temperature, top_p, top_k, etc.) are injected at + # runtime from the benchmark config's model_params block — absent here so + # the model's own defaults apply when not specified in model_params. diff --git a/src/inference_endpoint/evaluation/swebench_service/uv.lock b/src/inference_endpoint/evaluation/swebench_service/uv.lock new file mode 100644 index 000000000..1405f8ce4 --- /dev/null +++ b/src/inference_endpoint/evaluation/swebench_service/uv.lock @@ -0,0 +1,1594 @@ +version = 1 +revision = 3 +requires-python = "==3.12.*" +resolution-markers = [ + "sys_platform == 'win32'", + "sys_platform == 'emscripten'", + "sys_platform != 'emscripten' and sys_platform != 'win32'", +] + +[[package]] +name = "aiohappyeyeballs" +version = "2.7.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ce/f4/eec0465c2f67b2664688d0240b3212d5196fd89e741df67ddb81f8d35658/aiohappyeyeballs-2.7.1.tar.gz", hash = "sha256:065665c041c42a5938ed220bdcd7230f22527fbec085e1853d2402c8a3615d9d", size = 24757, upload-time = "2026-07-01T17:11:55.501Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/71/43/1947f06babed6b3f1d7f38b0c767f52df66bfb2bc10b468c4a7de9eceff2/aiohappyeyeballs-2.7.1-py3-none-any.whl", hash = "sha256:9243213661e29250eb41368e5daa826fc017156c3b8a11440826b2e3ed376472", size = 15038, upload-time = "2026-07-01T17:11:54.055Z" }, +] + +[[package]] +name = "aiohttp" +version = "3.14.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "aiohappyeyeballs" }, + { name = "aiosignal" }, + { name = "attrs" }, + { name = "frozenlist" }, + { name = "multidict" }, + { name = "propcache" }, + { name = "typing-extensions" }, + { name = "yarl" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/82/78/8ea7308cac6934de8c74a14f3d5f65d1c89287426688be79538d0e5c013d/aiohttp-3.14.1.tar.gz", hash = "sha256:307f2cff90a764d329e77040603fa032db89c5c24fdad50c4c15334cba744035", size = 7955794, upload-time = "2026-06-07T21:09:35.529Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1d/21/151624b51cd92553d95424daf4bf19f19ce9be9002d19253e7e7ce67197b/aiohttp-3.14.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:d35143e27778b4bb0fb189562d7f275bff79c62ab8e98459717c0ea617ff2480", size = 757402, upload-time = "2026-06-07T21:06:40.311Z" }, + { url = "https://files.pythonhosted.org/packages/c2/82/280619e0bd7bf2454987e19282616e84762255dd9c8468f62382e8c191f1/aiohttp-3.14.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:bcfb80a2cc36fba2534e5e5b5264dc7ae6fcd9bf15256da3e53d2f499e6fa29d", size = 512310, upload-time = "2026-06-07T21:06:42.207Z" }, + { url = "https://files.pythonhosted.org/packages/55/b2/2aac325583aaa1353045f96dffa586d8a34e8322e14a7ba49cffeb103ab4/aiohttp-3.14.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:27fd7c91e51729b4f7e1577865fa6d34c9adccbc39aabe9000285b48af9f0ec2", size = 512448, upload-time = "2026-06-07T21:06:43.813Z" }, + { url = "https://files.pythonhosted.org/packages/8a/72/a60607cb849faa8af8a356c9329ea2eb6f395d49e82cc82ccba1fd8deb8f/aiohttp-3.14.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:64c567bf9eaf664280116a8688f63016e6b32db2505908e2bdaca1b6438142f2", size = 1766854, upload-time = "2026-06-07T21:06:45.391Z" }, + { url = "https://files.pythonhosted.org/packages/b5/d3/d9fe1c9ec7557ab4d0d82bebaa728c6418f0b93295ec2f4ab015f7710cc7/aiohttp-3.14.1-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:f5e6ff2bdbb8f4cd3fbe41f99e25bbcd58e3bf9f13d3dd31a11e7917251cc77a", size = 1740884, upload-time = "2026-06-07T21:06:47.413Z" }, + { url = "https://files.pythonhosted.org/packages/c1/dc/f2cecfaf9337ba3e63f181500814ff502aa3d00d9c7ec93a9d23d10a27b2/aiohttp-3.14.1-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:2f73e01dc37122325caf079982621262f96d74823c179038a82fddfc50359264", size = 1810034, upload-time = "2026-06-07T21:06:50.165Z" }, + { url = "https://files.pythonhosted.org/packages/66/d7/2ff65c5e65c0d7476daf7e15c032e0805e36811185b9623e3238ad6c763e/aiohttp-3.14.1-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:bb2c0c80d431c0d03f2c7dbf125150fedd4f0de17366a7ca33f7ccb822391842", size = 1904054, upload-time = "2026-06-07T21:06:52.035Z" }, + { url = "https://files.pythonhosted.org/packages/20/9c/d445818389df371f56d141d881153ba23183c4735a03f7356ffb43f7757d/aiohttp-3.14.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3e6fc1a85fa7194a1a7d19f44e8609180f4a8eb5fa4c7ed8b4355f080fad235c", size = 1790278, upload-time = "2026-06-07T21:06:54.049Z" }, + { url = "https://files.pythonhosted.org/packages/4d/aa/bf04cb4d865fc6101c2229a294ad744973b72e513fdc5a6b791e6983d72a/aiohttp-3.14.1-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:686b6c0d3911ec387b444ddf5dc62fb7f7c0a7d5186a7861626496a5ab4aff95", size = 1591795, upload-time = "2026-06-07T21:06:55.911Z" }, + { url = "https://files.pythonhosted.org/packages/dc/b4/4dac0038960427ba832f6609dfb4ea5437d7fd80c72001b9e48f834f428b/aiohttp-3.14.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:c6fa4dc7ad6f8109c70bb1499e589f76b0b792baf39f9b017eb92c8a81d0a199", size = 1728397, upload-time = "2026-06-07T21:06:57.777Z" }, + { url = "https://files.pythonhosted.org/packages/2b/f9/7cd4e8ad7aa3b75f17d56bb5498dd604a93d4e6eece822ba0568c413fff0/aiohttp-3.14.1-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:87a5eea1b2a5e21e1ebdbb33ad4165359189327e63fc4e4894693e7f821ac817", size = 1766504, upload-time = "2026-06-07T21:07:00.009Z" }, + { url = "https://files.pythonhosted.org/packages/f9/df/fc01d9fcad0f73fed3f3d361f1f94f975947b50dff82919f6dc2bf4316cc/aiohttp-3.14.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:1c1421eb01d4fd608d88cc8290211d177a58532b55ad94076fb349c5bf467f0a", size = 1777806, upload-time = "2026-06-07T21:07:02.064Z" }, + { url = "https://files.pythonhosted.org/packages/41/09/47e2d090bddcc8fb4ccb4c314aadc32d7c5d9bb55f50f6ad1c92fc15d501/aiohttp-3.14.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:34b257ec41345c1e8f2df68fa908a7952f5de932723871eb633ecbbff396c9a4", size = 1580707, upload-time = "2026-06-07T21:07:03.942Z" }, + { url = "https://files.pythonhosted.org/packages/3d/36/f1a4ce904ae0b6930cfe9afc96d0896f7ec1a620c400405d63783bb95a9c/aiohttp-3.14.1-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:de538791a80e5d862addbc183f70f0158ac9b9bb872bb147f1fd2a683691e087", size = 1798121, upload-time = "2026-06-07T21:07:05.987Z" }, + { url = "https://files.pythonhosted.org/packages/70/0a/e0075ce9ca0279ee1d4f0c0b85f54fea02ebc83c3007651a72bece658fec/aiohttp-3.14.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:6f71173be42d3241d428f760122febb748de0623f44308a6f120d0dd9ec572e3", size = 1767580, upload-time = "2026-06-07T21:07:07.873Z" }, + { url = "https://files.pythonhosted.org/packages/3e/61/a0c0a8f327a9c52095cdd8e312391b00d3ed64ab6c72bb5c33d8ec251cf7/aiohttp-3.14.1-cp312-cp312-win32.whl", hash = "sha256:ec8dc383ee57ea3e883477dcca3f11b65d58199f1080acaf4cd6ad9a99698be4", size = 452771, upload-time = "2026-06-07T21:07:09.669Z" }, + { url = "https://files.pythonhosted.org/packages/df/d9/ea367c75f16ac9c6cdc8febb25e8318fa21a2b1bc8d6514d4b2d890bface/aiohttp-3.14.1-cp312-cp312-win_amd64.whl", hash = "sha256:2aa92c87868cd13674989f9ee83e5f9f7ea4237589b728048e1f0c8f6caa3271", size = 479873, upload-time = "2026-06-07T21:07:11.538Z" }, + { url = "https://files.pythonhosted.org/packages/03/64/8d96784a7851156db8a4c6c3f6f91042fdf39fb15a4cc38c8b3c14833c45/aiohttp-3.14.1-cp312-cp312-win_arm64.whl", hash = "sha256:2c840c90759922cb5e6dda94596e079a30fb5a5ba548e7e0dc00574703940847", size = 448073, upload-time = "2026-06-07T21:07:13.637Z" }, +] + +[[package]] +name = "aiosignal" +version = "1.4.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "frozenlist" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/61/62/06741b579156360248d1ec624842ad0edf697050bbaf7c3e46394e106ad1/aiosignal-1.4.0.tar.gz", hash = "sha256:f47eecd9468083c2029cc99945502cb7708b082c232f9aca65da147157b251c7", size = 25007, upload-time = "2025-07-03T22:54:43.528Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fb/76/641ae371508676492379f16e2fa48f4e2c11741bd63c48be4b12a6b09cba/aiosignal-1.4.0-py3-none-any.whl", hash = "sha256:053243f8b92b990551949e63930a839ff0cf0b0ebbe0597b0f3fb19e1a0fe82e", size = 7490, upload-time = "2025-07-03T22:54:42.156Z" }, +] + +[[package]] +name = "annotated-doc" +version = "0.0.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/57/ba/046ceea27344560984e26a590f90bc7f4a75b06701f653222458922b558c/annotated_doc-0.0.4.tar.gz", hash = "sha256:fbcda96e87e9c92ad167c2e53839e57503ecfda18804ea28102353485033faa4", size = 7288, upload-time = "2025-11-10T22:07:42.062Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1e/d3/26bf1008eb3d2daa8ef4cacc7f3bfdc11818d111f7e2d0201bc6e3b49d45/annotated_doc-0.0.4-py3-none-any.whl", hash = "sha256:571ac1dc6991c450b25a9c2d84a3705e2ae7a53467b5d111c24fa8baabbed320", size = 5303, upload-time = "2025-11-10T22:07:40.673Z" }, +] + +[[package]] +name = "annotated-types" +version = "0.7.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ee/67/531ea369ba64dcff5ec9c3402f9f51bf748cec26dde048a2f973a4eea7f5/annotated_types-0.7.0.tar.gz", hash = "sha256:aff07c09a53a08bc8cfccb9c85b05f1aa9a2a6f23728d790723543408344ce89", size = 16081, upload-time = "2024-05-20T21:33:25.928Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl", hash = "sha256:1f02e8b43a8fbbc3f3e0d4f0f4bfc8131bcb4eebe8849b8e5c773f3a1c582a53", size = 13643, upload-time = "2024-05-20T21:33:24.1Z" }, +] + +[[package]] +name = "anyio" +version = "4.14.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "idna" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/61/cc/a381afa6efea9f496eff839d4a6a1aed3bfafc7b3ab4b0d1b243a12573dd/anyio-4.14.2.tar.gz", hash = "sha256:cfa139f3ed1a23ee8f88a145ddb5ac7605b8bbfd8592baacd7ce3d8bb4313c7f", size = 260176, upload-time = "2026-07-12T20:29:07.082Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/da/35/f2287558c17e29fafc8ef3daf819bb9834061cfa43bff8014f7df7f63bdc/anyio-4.14.2-py3-none-any.whl", hash = "sha256:9f505dda5ac9f0c8309b5e8bd445a8c2bf7246f3ce950121e45ea15bc41d1494", size = 125813, upload-time = "2026-07-12T20:29:05.763Z" }, +] + +[[package]] +name = "attrs" +version = "26.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/9a/8e/82a0fe20a541c03148528be8cac2408564a6c9a0cc7e9171802bc1d26985/attrs-26.1.0.tar.gz", hash = "sha256:d03ceb89cb322a8fd706d4fb91940737b6642aa36998fe130a9bc96c985eff32", size = 952055, upload-time = "2026-03-19T14:22:25.026Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/64/b4/17d4b0b2a2dc85a6df63d1157e028ed19f90d4cd97c36717afef2bc2f395/attrs-26.1.0-py3-none-any.whl", hash = "sha256:c647aa4a12dfbad9333ca4e71fe62ddc36f4e63b2d260a37a8b83d2f043ac309", size = 67548, upload-time = "2026-03-19T14:22:23.645Z" }, +] + +[[package]] +name = "beautifulsoup4" +version = "4.15.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "soupsieve" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/43/65/318323f98dbee45d42dff61d8f047181bc6f2268a9068cfad035a46be5af/beautifulsoup4-4.15.0.tar.gz", hash = "sha256:288e3ca7d54b06f2ac191970bc275c1939cb46d450b255bf6718b04aa37ab4f7", size = 632571, upload-time = "2026-06-07T16:44:20.453Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/88/c6/92fcd42f1ba33e1184263f25bfabf3d27c383410470f169e4b8163bf9c17/beautifulsoup4-4.15.0-py3-none-any.whl", hash = "sha256:d6f88de62e1d4e38ecb1077eb9724cd0eff29d2a08ca16a401e9b9e93f117cf9", size = 109924, upload-time = "2026-06-07T16:44:21.566Z" }, +] + +[[package]] +name = "cbor2" +version = "6.1.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/3a/6f/07b4af8da8bd27f640362b1ac8271d80895407f2ede0c2bcc9433c06e1ca/cbor2-6.1.3.tar.gz", hash = "sha256:8d70680acb55c04ea5b5ad86da094f9612b53d5a8a65d0f5b3aafc3ce917ecbb", size = 89503, upload-time = "2026-07-04T10:36:48.793Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/31/16/cff14259c3d19a7f0ae88b6996fe4c85f6ff1764dad889ac8a39e843e39c/cbor2-6.1.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:3d939f55097c21e032f5a2d67592fcc57298986281f219356e2f519e4466f4ea", size = 412779, upload-time = "2026-07-04T10:36:04.975Z" }, + { url = "https://files.pythonhosted.org/packages/50/6c/f3641d19b7b85a63cb2756c10164131489c2cb46b379ec51ae22283fefb9/cbor2-6.1.3-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:b025009478d644dab407164fd60e3ef4381af284f5af6966df94c663756d949e", size = 457781, upload-time = "2026-07-04T10:36:06.349Z" }, + { url = "https://files.pythonhosted.org/packages/55/85/0c55a66f3037056bfb8e1c7184168085fdea67ae5830404498bcf466233b/cbor2-6.1.3-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:2226d32e102e375737656ad5d141ad8c6ae3e705e04e263f24756f0eb379c6c1", size = 468373, upload-time = "2026-07-04T10:36:07.769Z" }, + { url = "https://files.pythonhosted.org/packages/46/74/40f7db3e0d880560193916a5c9b744fcf299558bed7113f77c28237c7c29/cbor2-6.1.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:e61d465244d66ffed36492eef3b44d43795d76a2bba0663a2f15c186af7f7513", size = 523844, upload-time = "2026-07-04T10:36:09.404Z" }, + { url = "https://files.pythonhosted.org/packages/b5/a1/b5e07d6a08441c3a552fe2ae48ccb7e9dfc5065b9f6a3bae9879b4f0fbc0/cbor2-6.1.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:87fe7be8fab6ec4796aa127c1a52e09e79dbafd2aa31caf809cf04b8080a5975", size = 536238, upload-time = "2026-07-04T10:36:10.914Z" }, + { url = "https://files.pythonhosted.org/packages/c9/99/e166be0fd74bf3a91f5a0d103e34883efbc438d970f72cc8200e274787e5/cbor2-6.1.3-cp312-cp312-win32.whl", hash = "sha256:da25d345f01e6a40b2e5c57ef96b4dcff7be69394fb62f0f70e07f437f2376a9", size = 279858, upload-time = "2026-07-04T10:36:12.247Z" }, + { url = "https://files.pythonhosted.org/packages/d2/1b/90b4a121e40aba189c55a5822dd3c698eaf487e1d4a780ab18c804a5ef1c/cbor2-6.1.3-cp312-cp312-win_amd64.whl", hash = "sha256:d5514f693db6fa6f433b4096e9b604e6a7bf151c9ef1d2db86d0858e4c5e768f", size = 300929, upload-time = "2026-07-04T10:36:13.564Z" }, + { url = "https://files.pythonhosted.org/packages/19/db/52c58a8d33464927389dde8103997b3fa51b081ce29b347ac2cc4fd0dfbf/cbor2-6.1.3-cp312-cp312-win_arm64.whl", hash = "sha256:3d43183d7beb3d3cd198d69b31bd2ee487ed704a1150c75cb0a66d6ad63d8c1a", size = 290908, upload-time = "2026-07-04T10:36:14.857Z" }, +] + +[[package]] +name = "certifi" +version = "2026.6.17" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c9/c7/424b75da314c1045981bd9777432fad05a9e0c69daa4ed7e308bbaffe405/certifi-2026.6.17.tar.gz", hash = "sha256:024c88eeec92ca068db80f02b8b07c9cef7b9fe261d1d535abfd5abd6f6af432", size = 134594, upload-time = "2026-06-17T10:31:07.894Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ef/2f/c5464532e965badff2f4c4c1a3a83f5697f0d7c407ed0cda44aaa99bb451/certifi-2026.6.17-py3-none-any.whl", hash = "sha256:2227dcbaafe0d2f59279d1762ddddc37783ed4354594f194ffc31d20f41fc3db", size = 133289, upload-time = "2026-06-17T10:31:06.348Z" }, +] + +[[package]] +name = "cfgv" +version = "3.5.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/4e/b5/721b8799b04bf9afe054a3899c6cf4e880fcf8563cc71c15610242490a0c/cfgv-3.5.0.tar.gz", hash = "sha256:d5b1034354820651caa73ede66a6294d6e95c1b00acc5e9b098e917404669132", size = 7334, upload-time = "2025-11-19T20:55:51.612Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/db/3c/33bac158f8ab7f89b2e59426d5fe2e4f63f7ed25df84c036890172b412b5/cfgv-3.5.0-py2.py3-none-any.whl", hash = "sha256:a8dc6b26ad22ff227d2634a65cb388215ce6cc96bbcc5cfde7641ae87e8dacc0", size = 7445, upload-time = "2025-11-19T20:55:50.744Z" }, +] + +[[package]] +name = "chardet" +version = "7.4.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/19/b6/9df434a8eeba2e6628c465a1dfa31034228ef79b26f76f46278f4ef7e49d/chardet-7.4.3.tar.gz", hash = "sha256:cc1d4eb92a4ec1c2df3b490836ffa46922e599d34ce0bb75cf41fd2bf6303d56", size = 784800, upload-time = "2026-04-13T21:33:39.803Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/61/33/29de185079e6675c3f375546e30a559b7ddc75ce972f18d6e566cd9ea4eb/chardet-7.4.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:75d3c65cc16bddf40b8da1fd25ba84fca5f8070f2b14e86083653c1c85aee971", size = 874870, upload-time = "2026-04-13T21:33:05.977Z" }, + { url = "https://files.pythonhosted.org/packages/9c/2f/4c5af01fd1a7506a1d5375403d68925eac70289229492db5aa68b58103d8/chardet-7.4.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:29af5999f654e8729d251f1724a62b538b1262d9292cccaefddf8a02aae1ef6a", size = 854859, upload-time = "2026-04-13T21:33:07.381Z" }, + { url = "https://files.pythonhosted.org/packages/36/21/edb36ad5dfa48d7f8eed97ab43931ecdaa8c15166c21b1d614967e49d681/chardet-7.4.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:626f00299ad62dfe937058a09572beed442ccc7b58f87aa667949b20fd3db235", size = 875032, upload-time = "2026-04-13T21:33:08.741Z" }, + { url = "https://files.pythonhosted.org/packages/e5/59/a32a241d861cf180853a11c8e5a67641cb1b2af13c3a5ccce83ec07e2c9f/chardet-7.4.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9a4904dd5f071b7a7d7f50b4a67a86db3c902d243bf31708f1d5cde2f68239cb", size = 888283, upload-time = "2026-04-13T21:33:10.213Z" }, + { url = "https://files.pythonhosted.org/packages/87/2e/e1ee6a77abf3782c00e05b89c4d4328c8353bf9500661c4348df1dd68614/chardet-7.4.3-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:5d2879598bc220689e8ce509fe9c3f37ad2fca53a36be9c9bd91abdd91dd364f", size = 879974, upload-time = "2026-04-13T21:33:11.448Z" }, + { url = "https://files.pythonhosted.org/packages/32/60/fca69c534602a7ced04280c952a246ad1edde2a6ca3a164f65d32ac41fe7/chardet-7.4.3-cp312-cp312-win_amd64.whl", hash = "sha256:4b2799bd58e7245cfa8d4ab2e8ad1d76a5c3a5b1f32318eb6acca4c69a3e7101", size = 943973, upload-time = "2026-04-13T21:33:12.756Z" }, + { url = "https://files.pythonhosted.org/packages/8c/6c/0a40afdb50a0fe041ab95553b835a8160b6cf0e81edf2ae2fe9f5224cbf9/chardet-7.4.3-py3-none-any.whl", hash = "sha256:1173b74051570cf08099d7429d92e4882d375ad4217f92a6e5240ccfb26f231e", size = 626562, upload-time = "2026-04-13T21:33:38.559Z" }, +] + +[[package]] +name = "charset-normalizer" +version = "3.4.9" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/bd/2a/23f34ec9d04624958e137efdc394888716353190e75f25dd22c7a2c7a8aa/charset_normalizer-3.4.9.tar.gz", hash = "sha256:673611bbd43f0810bec0b0f028ddeaaa501190339cac411f347ac76917c3ae7b", size = 152439, upload-time = "2026-07-07T14:34:58.454Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/70/4a/ecbd131485c07fcdfad54e28946d513e3da22ef3b4bd854dcafae54ec739/charset_normalizer-3.4.9-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:45b0cc4e3556cd875e09102988d1ab8356c998b596c9fced84547c8138b487a0", size = 319300, upload-time = "2026-07-07T14:33:15.666Z" }, + { url = "https://files.pythonhosted.org/packages/ec/96/5d9364e3342d69f3a045e1777bc47c85c383e6e9466d561b33fdb419d1f9/charset_normalizer-3.4.9-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9b2aff1c7b3884512b9512c3eaadd9bab39fb45042ffaaa1dd08ff2b9f8109d9", size = 215802, upload-time = "2026-07-07T14:33:17.031Z" }, + { url = "https://files.pythonhosted.org/packages/4b/4c/5361f9aa7f2cb58d94f2ab831b3d493f69efb1d239654b4744e3c09527cb/charset_normalizer-3.4.9-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9104ed0bd76a429d46f9ec0dbc9b08ad1d2dcdf2b00a5a0daa1c145329b35b44", size = 237171, upload-time = "2026-07-07T14:33:18.576Z" }, + { url = "https://files.pythonhosted.org/packages/50/78/ce342ca4ff30b2eb49fe6d9578df85974f90c67d294113e94efdd9664cbd/charset_normalizer-3.4.9-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:7b86a2b16095d250c6f58b3d9b2eee6f4147754344f3dab0922f7c9bf7d226c9", size = 233075, upload-time = "2026-07-07T14:33:20.084Z" }, + { url = "https://files.pythonhosted.org/packages/01/c4/4fa4c8b3097a11f3c5f09a35b72ed6855fb1d332469504962ab7bafcc702/charset_normalizer-3.4.9-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5e226f6218febc71f6c1fc2fafb91c226f75bdc1d8fb12d66823716e891608fd", size = 224256, upload-time = "2026-07-07T14:33:21.747Z" }, + { url = "https://files.pythonhosted.org/packages/87/3a/ad914516df7e358a81aae018caa5e0470ba827fa6d763b1d2e87d920a5f6/charset_normalizer-3.4.9-cp312-cp312-manylinux_2_31_armv7l.whl", hash = "sha256:90c44bc373b7687f6948b693cceaea1348ae0975d7474746559494468e3c1d84", size = 208784, upload-time = "2026-07-07T14:33:23.313Z" }, + { url = "https://files.pythonhosted.org/packages/d7/74/3c12f9755717dfe5c5c87da63f35d765fa0c00382ec26bf23f7fae34f2ba/charset_normalizer-3.4.9-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:9cdef90ae47919cae358d8ab15797a800ed41da7aba5d72419fb510729e2ed4b", size = 219928, upload-time = "2026-07-07T14:33:24.814Z" }, + { url = "https://files.pythonhosted.org/packages/33/9a/895095b83e7907abd6d3d99aad3a38ad0d9686cc186cb0c94c24320fe63e/charset_normalizer-3.4.9-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:60f44ade2cf573dad7a277e6f8ca9a51a21dda572b13bd7d8539bb3cd5dbedde", size = 218489, upload-time = "2026-07-07T14:33:26.42Z" }, + { url = "https://files.pythonhosted.org/packages/a1/34/ef5c05f412f42520d7709b7d3784d19640839eb7366ded1755511585429f/charset_normalizer-3.4.9-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:a1786910334ed46ab1dd73222f2cd1e05c2c3bb39f6dddb4f8b36fc382058a39", size = 210267, upload-time = "2026-07-07T14:33:27.952Z" }, + { url = "https://files.pythonhosted.org/packages/83/dc/9b29fa4412b318bf3bfea985c35d67eb55e04b59a7c3f2237168b0e0be6f/charset_normalizer-3.4.9-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:03d07803992c6c7bbc976327f34b18b6160327fc81cb82c9d504720ac0be3b62", size = 226030, upload-time = "2026-07-07T14:33:29.397Z" }, + { url = "https://files.pythonhosted.org/packages/0e/42/6dbc00b8cd16011691203e33570fa42ed5746599a2e878112d16eab403a3/charset_normalizer-3.4.9-cp312-cp312-win32.whl", hash = "sha256:78841cccf1af7b40f6f716338d50c0902dbe88d9f800b3c973b7a9a0a693a642", size = 151185, upload-time = "2026-07-07T14:33:30.781Z" }, + { url = "https://files.pythonhosted.org/packages/80/cc/f920afd1a23c58ccd53c1d36085a71893a4737ff5e66e0371efab6809850/charset_normalizer-3.4.9-cp312-cp312-win_amd64.whl", hash = "sha256:4b3dac63058cc36820b0dd072f89898604e2d39686fe05321729d00d8ac185a0", size = 162557, upload-time = "2026-07-07T14:33:32.176Z" }, + { url = "https://files.pythonhosted.org/packages/f0/e6/0386d43a261ff4e4b30c5857af7df877254b46bec7b9d1b74b6bf969a90b/charset_normalizer-3.4.9-cp312-cp312-win_arm64.whl", hash = "sha256:78fa18e436a1a0e58dbd7e02fc4473f3f32cceb12df9dfca542d075961c307d2", size = 152665, upload-time = "2026-07-07T14:33:33.711Z" }, + { url = "https://files.pythonhosted.org/packages/98/2b/f97f1c193fb855c345d678f5077d6926034db0722df74c8f057020e05a25/charset_normalizer-3.4.9-py3-none-any.whl", hash = "sha256:68e5f26a1ad57ded6d1cfb85331d1c1a195314756471d97758c48498bb4dcdf5", size = 64538, upload-time = "2026-07-07T14:34:56.993Z" }, +] + +[[package]] +name = "click" +version = "8.4.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/76/d4/81420972a676e8ffea40450d8c8c92943e7218a78fe9b64359836cc9876b/click-8.4.2.tar.gz", hash = "sha256:9a6cea6e60b17ebe0a44c5cc636d94f09bd66142c1cd7d8b4cd731c4917a15f6", size = 338000, upload-time = "2026-06-24T17:45:15.148Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fb/e2/79c688af8b210d232694e31e59da9f6ec747bae31c3f5946e4e9b98860d5/click-8.4.2-py3-none-any.whl", hash = "sha256:e6f9f66136c816745b9d65817da91d61d957fb16e02e4dcd0552553c5a197b76", size = 119243, upload-time = "2026-06-24T17:45:13.73Z" }, +] + +[[package]] +name = "colorama" +version = "0.4.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, +] + +[[package]] +name = "datasets" +version = "5.0.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "dill" }, + { name = "filelock" }, + { name = "fsspec", extra = ["http"] }, + { name = "httpx" }, + { name = "huggingface-hub" }, + { name = "multiprocess" }, + { name = "numpy" }, + { name = "packaging" }, + { name = "pandas" }, + { name = "pyarrow" }, + { name = "pyyaml" }, + { name = "requests" }, + { name = "tqdm" }, + { name = "xxhash" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/d9/85/ce4f780c32f7e36d71257f1c27e8ba898ebe379cb54f211f5f2013f2c219/datasets-5.0.0.tar.gz", hash = "sha256:83dbbbdb07a33b82192b8c419deb18739b138ee2ce1a322d55ce6b100954ec1a", size = 631708, upload-time = "2026-06-05T13:18:26.124Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/05/66/73034ad30b59f13439b75e620989dacba4c047256e358ba7c2e9ec98ea22/datasets-5.0.0-py3-none-any.whl", hash = "sha256:7dd34927a0fd7046e98aad5cb9430e699c373238a15befa7b9bf22b991a7fee6", size = 555084, upload-time = "2026-06-05T13:18:24.435Z" }, +] + +[[package]] +name = "dill" +version = "0.4.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/81/e1/56027a71e31b02ddc53c7d65b01e68edf64dea2932122fe7746a516f75d5/dill-0.4.1.tar.gz", hash = "sha256:423092df4182177d4d8ba8290c8a5b640c66ab35ec7da59ccfa00f6fa3eea5fa", size = 187315, upload-time = "2026-01-19T02:36:56.85Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1e/77/dc8c558f7593132cf8fefec57c4f60c83b16941c574ac5f619abb3ae7933/dill-0.4.1-py3-none-any.whl", hash = "sha256:1e1ce33e978ae97fcfcff5638477032b801c46c7c65cf717f95fbc2248f79a9d", size = 120019, upload-time = "2026-01-19T02:36:55.663Z" }, +] + +[[package]] +name = "distlib" +version = "0.4.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c9/02/bd72be9134d25ed783ecbbc38a539ffaefbf90c78418c7fb7229600dbac7/distlib-0.4.3.tar.gz", hash = "sha256:f152097224a0ae24be5a0f6bae1b9359af82133bce63f98a95f86cae1aede9ed", size = 615141, upload-time = "2026-06-12T08:04:52.847Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/02/08/9c41fb51ab5b43eb21674aff13df270e8ba6c4b29c8624e328dc7a9482af/distlib-0.4.3-py2.py3-none-any.whl", hash = "sha256:4b0ce306c966eb73bc3a7b6abad017c556dadd92c44701562cd528ac7fde4d5b", size = 470628, upload-time = "2026-06-12T08:04:50.506Z" }, +] + +[[package]] +name = "distro" +version = "1.9.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/fc/f8/98eea607f65de6527f8a2e8885fc8015d3e6f5775df186e443e0964a11c3/distro-1.9.0.tar.gz", hash = "sha256:2fa77c6fd8940f116ee1d6b94a2f90b13b5ea8d019b98bc8bafdcabcdd9bdbed", size = 60722, upload-time = "2023-12-24T09:54:32.31Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/12/b3/231ffd4ab1fc9d679809f356cebee130ac7daa00d6d6f3206dd4fd137e9e/distro-1.9.0-py3-none-any.whl", hash = "sha256:7bffd925d65168f85027d8da9af6bddab658135b840670a223589bc0c8ef02b2", size = 20277, upload-time = "2023-12-24T09:54:30.421Z" }, +] + +[[package]] +name = "docker" +version = "7.2.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "pywin32", marker = "sys_platform == 'win32'" }, + { name = "requests" }, + { name = "urllib3" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/88/7f/731ff914b0255d3d065f45fd4e626d4b8c95dbcbaada049f337a6ac16410/docker-7.2.0.tar.gz", hash = "sha256:cebb93773d334f778e023a7ee352a8d6e13ab1bd3b863a4d4a59dec897df43ac", size = 118731, upload-time = "2026-07-09T14:53:46.39Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/75/23/529140fe1aab80fc6992f93a706deec709140a6397439139a054e1515c45/docker-7.2.0-py3-none-any.whl", hash = "sha256:a3f45fdeb9165e2d25d9a1d02ddf3bc70fb572cf5ebbf9b58558c22caf29b71f", size = 148775, upload-time = "2026-07-09T14:53:45.224Z" }, +] + +[[package]] +name = "fastcore" +version = "2.0.5" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c7/83/a4452a9e0c078d844e0882745d0a44d3f57c968e1ae0d5dbc06892a403d0/fastcore-2.0.5.tar.gz", hash = "sha256:7be70ec5517723e4caeac0725a75942f9f081d152470d1401699a1ba3f536c01", size = 103271, upload-time = "2026-07-13T07:19:35.999Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5e/cc/4bc2c4514444fe27514700a4fb2d8385f32cb65f3f0e0cf752eceed5854a/fastcore-2.0.5-py3-none-any.whl", hash = "sha256:88a7149aea4457717a02bdc74a4e3e4669d1cc9b611759da984333f83a280d85", size = 108123, upload-time = "2026-07-13T07:19:34.53Z" }, +] + +[[package]] +name = "fastspec" +version = "0.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "fastcore" }, + { name = "httpx" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/aa/aa/e902620ee18e400802d1488a9caca9ba3af68d3d2ebf2fa250fb4673421c/fastspec-0.1.0.tar.gz", hash = "sha256:b81a01ecdf1f8995eb6c25eb32a3f8dabbd37c9a238432c907b88449590b269d", size = 27268, upload-time = "2026-07-08T01:07:16.698Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d8/7b/5c88b14a72608a4c259c7d6cfa63fe7124fc3129c74cecf107c7797b157d/fastspec-0.1.0-py3-none-any.whl", hash = "sha256:354e96acf7b6eafae9ef284aa7b3a4c08742d545068544eac16b7d177fe02d7a", size = 25969, upload-time = "2026-07-08T01:07:15.59Z" }, +] + +[[package]] +name = "fastuuid" +version = "0.14.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c3/7d/d9daedf0f2ebcacd20d599928f8913e9d2aea1d56d2d355a93bfa2b611d7/fastuuid-0.14.0.tar.gz", hash = "sha256:178947fc2f995b38497a74172adee64fdeb8b7ec18f2a5934d037641ba265d26", size = 18232, upload-time = "2025-10-19T22:19:22.402Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/02/a2/e78fcc5df65467f0d207661b7ef86c5b7ac62eea337c0c0fcedbeee6fb13/fastuuid-0.14.0-cp312-cp312-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:77e94728324b63660ebf8adb27055e92d2e4611645bf12ed9d88d30486471d0a", size = 510164, upload-time = "2025-10-19T22:31:45.635Z" }, + { url = "https://files.pythonhosted.org/packages/2b/b3/c846f933f22f581f558ee63f81f29fa924acd971ce903dab1a9b6701816e/fastuuid-0.14.0-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:caa1f14d2102cb8d353096bc6ef6c13b2c81f347e6ab9d6fbd48b9dea41c153d", size = 261837, upload-time = "2025-10-19T22:38:38.53Z" }, + { url = "https://files.pythonhosted.org/packages/54/ea/682551030f8c4fa9a769d9825570ad28c0c71e30cf34020b85c1f7ee7382/fastuuid-0.14.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:d23ef06f9e67163be38cece704170486715b177f6baae338110983f99a72c070", size = 251370, upload-time = "2025-10-19T22:40:26.07Z" }, + { url = "https://files.pythonhosted.org/packages/14/dd/5927f0a523d8e6a76b70968e6004966ee7df30322f5fc9b6cdfb0276646a/fastuuid-0.14.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0c9ec605ace243b6dbe3bd27ebdd5d33b00d8d1d3f580b39fdd15cd96fd71796", size = 277766, upload-time = "2025-10-19T22:37:23.779Z" }, + { url = "https://files.pythonhosted.org/packages/16/6e/c0fb547eef61293153348f12e0f75a06abb322664b34a1573a7760501336/fastuuid-0.14.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:808527f2407f58a76c916d6aa15d58692a4a019fdf8d4c32ac7ff303b7d7af09", size = 278105, upload-time = "2025-10-19T22:26:56.821Z" }, + { url = "https://files.pythonhosted.org/packages/2d/b1/b9c75e03b768f61cf2e84ee193dc18601aeaf89a4684b20f2f0e9f52b62c/fastuuid-0.14.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:2fb3c0d7fef6674bbeacdd6dbd386924a7b60b26de849266d1ff6602937675c8", size = 301564, upload-time = "2025-10-19T22:30:31.604Z" }, + { url = "https://files.pythonhosted.org/packages/fc/fa/f7395fdac07c7a54f18f801744573707321ca0cee082e638e36452355a9d/fastuuid-0.14.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:ab3f5d36e4393e628a4df337c2c039069344db5f4b9d2a3c9cea48284f1dd741", size = 459659, upload-time = "2025-10-19T22:31:32.341Z" }, + { url = "https://files.pythonhosted.org/packages/66/49/c9fd06a4a0b1f0f048aacb6599e7d96e5d6bc6fa680ed0d46bf111929d1b/fastuuid-0.14.0-cp312-cp312-musllinux_1_1_i686.whl", hash = "sha256:b9a0ca4f03b7e0b01425281ffd44e99d360e15c895f1907ca105854ed85e2057", size = 478430, upload-time = "2025-10-19T22:26:22.962Z" }, + { url = "https://files.pythonhosted.org/packages/be/9c/909e8c95b494e8e140e8be6165d5fc3f61fdc46198c1554df7b3e1764471/fastuuid-0.14.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:3acdf655684cc09e60fb7e4cf524e8f42ea760031945aa8086c7eae2eeeabeb8", size = 450894, upload-time = "2025-10-19T22:27:01.647Z" }, + { url = "https://files.pythonhosted.org/packages/90/eb/d29d17521976e673c55ef7f210d4cdd72091a9ec6755d0fd4710d9b3c871/fastuuid-0.14.0-cp312-cp312-win32.whl", hash = "sha256:9579618be6280700ae36ac42c3efd157049fe4dd40ca49b021280481c78c3176", size = 154374, upload-time = "2025-10-19T22:29:19.879Z" }, + { url = "https://files.pythonhosted.org/packages/cc/fc/f5c799a6ea6d877faec0472d0b27c079b47c86b1cdc577720a5386483b36/fastuuid-0.14.0-cp312-cp312-win_amd64.whl", hash = "sha256:d9e4332dc4ba054434a9594cbfaf7823b57993d7d8e7267831c3e059857cf397", size = 156550, upload-time = "2025-10-19T22:27:49.658Z" }, +] + +[[package]] +name = "filelock" +version = "3.29.7" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/35/94/00f2059e4835eace3ae8fde680b932c496f8ec7bdc99168dfa53fb2e6b79/filelock-3.29.7.tar.gz", hash = "sha256:5b481979797ae69e72f0b389d89a80bdd585c260c5b3f1fb9c0a5ba9bb3f195d", size = 71521, upload-time = "2026-07-08T05:46:58.716Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/60/02/be4a57b60c7149b55b9e3b3c13f609cd8eb5307c751f22bd8fb8d262e75b/filelock-3.29.7-py3-none-any.whl", hash = "sha256:987db6f789a3a2a59f55081801b2b3697cb97e2a736b5f1a9e99b559285fbc51", size = 46036, upload-time = "2026-07-08T05:46:57.53Z" }, +] + +[[package]] +name = "frozenlist" +version = "1.8.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/2d/f5/c831fac6cc817d26fd54c7eaccd04ef7e0288806943f7cc5bbf69f3ac1f0/frozenlist-1.8.0.tar.gz", hash = "sha256:3ede829ed8d842f6cd48fc7081d7a41001a56f1f38603f9d49bf3020d59a31ad", size = 45875, upload-time = "2025-10-06T05:38:17.865Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/69/29/948b9aa87e75820a38650af445d2ef2b6b8a6fab1a23b6bb9e4ef0be2d59/frozenlist-1.8.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:78f7b9e5d6f2fdb88cdde9440dc147259b62b9d3b019924def9f6478be254ac1", size = 87782, upload-time = "2025-10-06T05:36:06.649Z" }, + { url = "https://files.pythonhosted.org/packages/64/80/4f6e318ee2a7c0750ed724fa33a4bdf1eacdc5a39a7a24e818a773cd91af/frozenlist-1.8.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:229bf37d2e4acdaf808fd3f06e854a4a7a3661e871b10dc1f8f1896a3b05f18b", size = 50594, upload-time = "2025-10-06T05:36:07.69Z" }, + { url = "https://files.pythonhosted.org/packages/2b/94/5c8a2b50a496b11dd519f4a24cb5496cf125681dd99e94c604ccdea9419a/frozenlist-1.8.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f833670942247a14eafbb675458b4e61c82e002a148f49e68257b79296e865c4", size = 50448, upload-time = "2025-10-06T05:36:08.78Z" }, + { url = "https://files.pythonhosted.org/packages/6a/bd/d91c5e39f490a49df14320f4e8c80161cfcce09f1e2cde1edd16a551abb3/frozenlist-1.8.0-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:494a5952b1c597ba44e0e78113a7266e656b9794eec897b19ead706bd7074383", size = 242411, upload-time = "2025-10-06T05:36:09.801Z" }, + { url = "https://files.pythonhosted.org/packages/8f/83/f61505a05109ef3293dfb1ff594d13d64a2324ac3482be2cedc2be818256/frozenlist-1.8.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:96f423a119f4777a4a056b66ce11527366a8bb92f54e541ade21f2374433f6d4", size = 243014, upload-time = "2025-10-06T05:36:11.394Z" }, + { url = "https://files.pythonhosted.org/packages/d8/cb/cb6c7b0f7d4023ddda30cf56b8b17494eb3a79e3fda666bf735f63118b35/frozenlist-1.8.0-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:3462dd9475af2025c31cc61be6652dfa25cbfb56cbbf52f4ccfe029f38decaf8", size = 234909, upload-time = "2025-10-06T05:36:12.598Z" }, + { url = "https://files.pythonhosted.org/packages/31/c5/cd7a1f3b8b34af009fb17d4123c5a778b44ae2804e3ad6b86204255f9ec5/frozenlist-1.8.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c4c800524c9cd9bac5166cd6f55285957fcfc907db323e193f2afcd4d9abd69b", size = 250049, upload-time = "2025-10-06T05:36:14.065Z" }, + { url = "https://files.pythonhosted.org/packages/c0/01/2f95d3b416c584a1e7f0e1d6d31998c4a795f7544069ee2e0962a4b60740/frozenlist-1.8.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:d6a5df73acd3399d893dafc71663ad22534b5aa4f94e8a2fabfe856c3c1b6a52", size = 256485, upload-time = "2025-10-06T05:36:15.39Z" }, + { url = "https://files.pythonhosted.org/packages/ce/03/024bf7720b3abaebcff6d0793d73c154237b85bdf67b7ed55e5e9596dc9a/frozenlist-1.8.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:405e8fe955c2280ce66428b3ca55e12b3c4e9c336fb2103a4937e891c69a4a29", size = 237619, upload-time = "2025-10-06T05:36:16.558Z" }, + { url = "https://files.pythonhosted.org/packages/69/fa/f8abdfe7d76b731f5d8bd217827cf6764d4f1d9763407e42717b4bed50a0/frozenlist-1.8.0-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:908bd3f6439f2fef9e85031b59fd4f1297af54415fb60e4254a95f75b3cab3f3", size = 250320, upload-time = "2025-10-06T05:36:17.821Z" }, + { url = "https://files.pythonhosted.org/packages/f5/3c/b051329f718b463b22613e269ad72138cc256c540f78a6de89452803a47d/frozenlist-1.8.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:294e487f9ec720bd8ffcebc99d575f7eff3568a08a253d1ee1a0378754b74143", size = 246820, upload-time = "2025-10-06T05:36:19.046Z" }, + { url = "https://files.pythonhosted.org/packages/0f/ae/58282e8f98e444b3f4dd42448ff36fa38bef29e40d40f330b22e7108f565/frozenlist-1.8.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:74c51543498289c0c43656701be6b077f4b265868fa7f8a8859c197006efb608", size = 250518, upload-time = "2025-10-06T05:36:20.763Z" }, + { url = "https://files.pythonhosted.org/packages/8f/96/007e5944694d66123183845a106547a15944fbbb7154788cbf7272789536/frozenlist-1.8.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:776f352e8329135506a1d6bf16ac3f87bc25b28e765949282dcc627af36123aa", size = 239096, upload-time = "2025-10-06T05:36:22.129Z" }, + { url = "https://files.pythonhosted.org/packages/66/bb/852b9d6db2fa40be96f29c0d1205c306288f0684df8fd26ca1951d461a56/frozenlist-1.8.0-cp312-cp312-win32.whl", hash = "sha256:433403ae80709741ce34038da08511d4a77062aa924baf411ef73d1146e74faf", size = 39985, upload-time = "2025-10-06T05:36:23.661Z" }, + { url = "https://files.pythonhosted.org/packages/b8/af/38e51a553dd66eb064cdf193841f16f077585d4d28394c2fa6235cb41765/frozenlist-1.8.0-cp312-cp312-win_amd64.whl", hash = "sha256:34187385b08f866104f0c0617404c8eb08165ab1272e884abc89c112e9c00746", size = 44591, upload-time = "2025-10-06T05:36:24.958Z" }, + { url = "https://files.pythonhosted.org/packages/a7/06/1dc65480ab147339fecc70797e9c2f69d9cea9cf38934ce08df070fdb9cb/frozenlist-1.8.0-cp312-cp312-win_arm64.whl", hash = "sha256:fe3c58d2f5db5fbd18c2987cba06d51b0529f52bc3a6cdc33d3f4eab725104bd", size = 40102, upload-time = "2025-10-06T05:36:26.333Z" }, + { url = "https://files.pythonhosted.org/packages/9a/9a/e35b4a917281c0b8419d4207f4334c8e8c5dbf4f3f5f9ada73958d937dcc/frozenlist-1.8.0-py3-none-any.whl", hash = "sha256:0c18a16eab41e82c295618a77502e17b195883241c563b00f0aa5106fc4eaa0d", size = 13409, upload-time = "2025-10-06T05:38:16.721Z" }, +] + +[[package]] +name = "fsspec" +version = "2026.4.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d5/8d/1c51c094345df128ca4a990d633fe1a0ff28726c9e6b3c41ba65087bba1d/fsspec-2026.4.0.tar.gz", hash = "sha256:301d8ac70ae90ef3ad05dcf94d6c3754a097f9b5fe4667d2787aa359ec7df7e4", size = 312760, upload-time = "2026-04-29T20:42:38.635Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d5/0c/043d5e551459da400957a1395e0febbf771446ff34291afcbe3d8be2a279/fsspec-2026.4.0-py3-none-any.whl", hash = "sha256:11ef7bb35dab8a394fde6e608221d5cf3e8499401c249bebaeaad760a1a8dec2", size = 203402, upload-time = "2026-04-29T20:42:36.842Z" }, +] + +[package.optional-dependencies] +http = [ + { name = "aiohttp" }, +] + +[[package]] +name = "ghapi" +version = "2.0.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "fastcore" }, + { name = "fastspec" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/80/c7/1c1f2a042f9ad88f4effd9b84198d7be01e7f283ffa86a2d801d2cd58180/ghapi-2.0.1.tar.gz", hash = "sha256:f371e2794fffae53e2d891758c2696a21ba4a44c843e9142006077bd62a88641", size = 173467, upload-time = "2026-07-11T04:31:49.355Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7f/dd/c43af3b2426dee9245acef314bf8a55d18d5d7c443394a663e822daf9341/ghapi-2.0.1-py3-none-any.whl", hash = "sha256:d82cd571826835780166995bd5217a2df339aab85ca7491ffcb1eedbf7950863", size = 172531, upload-time = "2026-07-11T04:31:47.668Z" }, +] + +[[package]] +name = "gitdb" +version = "4.0.12" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "smmap" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/72/94/63b0fc47eb32792c7ba1fe1b694daec9a63620db1e313033d18140c2320a/gitdb-4.0.12.tar.gz", hash = "sha256:5ef71f855d191a3326fcfbc0d5da835f26b13fbcba60c32c21091c349ffdb571", size = 394684, upload-time = "2025-01-02T07:20:46.413Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a0/61/5c78b91c3143ed5c14207f463aecfc8f9dbb5092fb2869baf37c273b2705/gitdb-4.0.12-py3-none-any.whl", hash = "sha256:67073e15955400952c6565cc3e707c554a4eea2e428946f7a4c162fab9bd9bcf", size = 62794, upload-time = "2025-01-02T07:20:43.624Z" }, +] + +[[package]] +name = "gitpython" +version = "3.1.51" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "gitdb" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/59/30/a8a0c15f9480dc91b5b7f11ebd26105e5f80898d7ff02da197fef35d8395/gitpython-3.1.51.tar.gz", hash = "sha256:22c9c94bb6b0b9f3c7157c684fece45a414cea204586b600beae6cd4570dcd6d", size = 223519, upload-time = "2026-07-12T13:40:07.922Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f8/11/1232bb1ba52a230f20ff08e1b3df7cd9d43a71b61cc0a1c37941064fe4c7/gitpython-3.1.51-py3-none-any.whl", hash = "sha256:d5b29685708f3c80a56db040b665bfd69492c8e150b5848532af8013b686c5a4", size = 215246, upload-time = "2026-07-12T13:40:06.43Z" }, +] + +[[package]] +name = "grpclib" +version = "0.4.9" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "h2" }, + { name = "multidict" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/5b/28/5a2c299ec82a876a252c5919aa895a6f1d1d35c96417c5ce4a4660dc3a80/grpclib-0.4.9.tar.gz", hash = "sha256:cc589c330fa81004c6400a52a566407574498cb5b055fa927013361e21466c46", size = 84798, upload-time = "2025-12-14T22:23:14.349Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5c/90/b0cbbd9efcc82816c58f31a34963071aa19fb792a212a5d9caf8e0fc3097/grpclib-0.4.9-py3-none-any.whl", hash = "sha256:7762ec1c8ed94dfad597475152dd35cbd11aecaaca2f243e29702435ca24cf0e", size = 77063, upload-time = "2025-12-14T22:23:13.224Z" }, +] + +[[package]] +name = "h11" +version = "0.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/01/ee/02a2c011bdab74c6fb3c75474d40b3052059d95df7e73351460c8588d963/h11-0.16.0.tar.gz", hash = "sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1", size = 101250, upload-time = "2025-04-24T03:35:25.427Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/04/4b/29cac41a4d98d144bf5f6d33995617b185d14b22401f75ca86f384e87ff1/h11-0.16.0-py3-none-any.whl", hash = "sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86", size = 37515, upload-time = "2025-04-24T03:35:24.344Z" }, +] + +[[package]] +name = "h2" +version = "4.3.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "hpack" }, + { name = "hyperframe" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/1d/17/afa56379f94ad0fe8defd37d6eb3f89a25404ffc71d4d848893d270325fc/h2-4.3.0.tar.gz", hash = "sha256:6c59efe4323fa18b47a632221a1888bd7fde6249819beda254aeca909f221bf1", size = 2152026, upload-time = "2025-08-23T18:12:19.778Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/69/b2/119f6e6dcbd96f9069ce9a2665e0146588dc9f88f29549711853645e736a/h2-4.3.0-py3-none-any.whl", hash = "sha256:c438f029a25f7945c69e0ccf0fb951dc3f73a5f6412981daee861431b70e2bdd", size = 61779, upload-time = "2025-08-23T18:12:17.779Z" }, +] + +[[package]] +name = "hf-xet" +version = "1.5.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/4b/2d/57fd21d84d93efb4bd0b962383790e19dd1bc053501b4264c97903b4e83e/hf_xet-1.5.1.tar.gz", hash = "sha256:51ef4500dab3764b41135ee1381a4b62ce56fc54d4c92b719b59e597d6df5bf6", size = 876636, upload-time = "2026-06-08T23:02:53.897Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7a/d8/5e54cf37434759d1f4f2ba9b66077ff9d4c4e1f37b6bd7975da5c40d94ab/hf_xet-1.5.1-cp37-abi3-macosx_10_12_x86_64.whl", hash = "sha256:6abd35c3221eff63836618ddfb954dcf84798603f71d8e33e3ed7b04acfdbe6e", size = 4077794, upload-time = "2026-06-08T23:02:40.656Z" }, + { url = "https://files.pythonhosted.org/packages/35/94/4b2ecfbad8f8b04701a23aefb62f540b9137d058b7e1dbef16a32676f0e9/hf_xet-1.5.1-cp37-abi3-macosx_11_0_arm64.whl", hash = "sha256:94e761bbd266bf4c03cee73753916062665ce8365aa40ed321f45afcb934b41e", size = 3845354, upload-time = "2026-06-08T23:02:42.702Z" }, + { url = "https://files.pythonhosted.org/packages/de/cc/f99f4bc7295023d7bd9ebbfd51f75cc530ca262c1227666268b8208f4b77/hf_xet-1.5.1-cp37-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:892e3a3a3aecc12aded8b93cf4f9cd059282c7de0732f7d55026f3abdf474350", size = 4514864, upload-time = "2026-06-08T23:02:44.497Z" }, + { url = "https://files.pythonhosted.org/packages/cd/6e/21f7e5a2381278bd3b7b7a5a4d90038518bb6308a0c1daf5d9f8268bb178/hf_xet-1.5.1-cp37-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:a93df2039190502835b1db8cd7e178b0b7b889fe9ab51299d5ced26e0dd879a4", size = 4303784, upload-time = "2026-06-08T23:02:46.203Z" }, + { url = "https://files.pythonhosted.org/packages/35/0e/f992bb6927ac1cb30ef74e62268f551f338bc32b2191f7c96a44c6f7283e/hf_xet-1.5.1-cp37-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:0c97106032ef70467b4f6bc2d0ccc266d7613ee076afc56516c502f87ce1c4a6", size = 4500703, upload-time = "2026-06-08T23:02:47.628Z" }, + { url = "https://files.pythonhosted.org/packages/fb/d1/90a498d05447980b977b1669246eeeeae4cfb0ea3e7a286eaba627f91bf9/hf_xet-1.5.1-cp37-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:6208adb15d192b90e4c2ad2a27ed864359b2cb0f2494eb6d7c7f3699ac02e2bf", size = 4719498, upload-time = "2026-06-08T23:02:49.268Z" }, + { url = "https://files.pythonhosted.org/packages/6d/b6/20f99cfe97cc663a711f7b33cc21d4793e51968e9a26125b4afcd77315ba/hf_xet-1.5.1-cp37-abi3-win_amd64.whl", hash = "sha256:f7b3002f95d1c13e24bcb4537baa8f0eb3838957067c91bb4959bc004a6435f5", size = 4026419, upload-time = "2026-06-08T23:02:50.829Z" }, + { url = "https://files.pythonhosted.org/packages/f9/fa/77453694888f03e5a8c8852d1514a0894d8e81c622d39edbaf308ea0dcf4/hf_xet-1.5.1-cp37-abi3-win_arm64.whl", hash = "sha256:93d090b57b211133f6c0dab0205ef5cb6d89162979ba75a74845045cc3063b8e", size = 3855178, upload-time = "2026-06-08T23:02:52.452Z" }, +] + +[[package]] +name = "hpack" +version = "4.2.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/26/5b/fcabf6028144a8723726318b07a32c2f3314acdff6265743cf08a344b18e/hpack-4.2.0.tar.gz", hash = "sha256:0895cfa3b5531fc65fe439c05eb65144f123bf7a394fcaa56aa423548d8e45c0", size = 51300, upload-time = "2026-06-23T18:34:46.667Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/71/b4/4a9fcfb2aef6ba44d9073ecd301443aa00b3dac95de5619f2a7de7ec8a91/hpack-4.2.0-py3-none-any.whl", hash = "sha256:858ac0b02280fa582b5080d68db0899c62a80375e0e5413a74970c5e518b6986", size = 34246, upload-time = "2026-06-23T18:34:45.472Z" }, +] + +[[package]] +name = "httpcore" +version = "1.0.9" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "certifi" }, + { name = "h11" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/06/94/82699a10bca87a5556c9c59b5963f2d039dbd239f25bc2a63907a05a14cb/httpcore-1.0.9.tar.gz", hash = "sha256:6e34463af53fd2ab5d807f399a9b45ea31c3dfa2276f15a2c3f00afff6e176e8", size = 85484, upload-time = "2025-04-24T22:06:22.219Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7e/f5/f66802a942d491edb555dd61e3a9961140fd64c90bce1eafd741609d334d/httpcore-1.0.9-py3-none-any.whl", hash = "sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55", size = 78784, upload-time = "2025-04-24T22:06:20.566Z" }, +] + +[[package]] +name = "httpx" +version = "0.28.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "certifi" }, + { name = "httpcore" }, + { name = "idna" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b1/df/48c586a5fe32a0f01324ee087459e112ebb7224f646c0b5023f5e79e9956/httpx-0.28.1.tar.gz", hash = "sha256:75e98c5f16b0f35b567856f597f06ff2270a374470a5c2392242528e3e3e42fc", size = 141406, upload-time = "2024-12-06T15:37:23.222Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2a/39/e50c7c3a983047577ee07d2a9e53faf5a69493943ec3f6a384bdc792deb2/httpx-0.28.1-py3-none-any.whl", hash = "sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad", size = 73517, upload-time = "2024-12-06T15:37:21.509Z" }, +] + +[[package]] +name = "huggingface-hub" +version = "1.23.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "click" }, + { name = "filelock" }, + { name = "fsspec" }, + { name = "hf-xet", marker = "platform_machine == 'AMD64' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'arm64' or platform_machine == 'x86_64'" }, + { name = "httpx" }, + { name = "packaging" }, + { name = "pyyaml" }, + { name = "tqdm" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/1a/8f/999e4dda11c6187c78f090eac00895a47e11a0049308f07579bcb7aa3aa2/huggingface_hub-1.23.0.tar.gz", hash = "sha256:c04997fb8bbdace1e57b7703d30ed7678af51f70d00d241819ff411b92ae9a88", size = 919163, upload-time = "2026-07-09T14:49:32.315Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f1/ce/13b2ba57838b8db1e6bd033c1b21ce0b9f6153b87d4e4939f77074e41eb0/huggingface_hub-1.23.0-py3-none-any.whl", hash = "sha256:b1d604788f5adc7f0eb246e03e0ec19011ca06e38400218c347dccc3dffa64a2", size = 770336, upload-time = "2026-07-09T14:49:30.597Z" }, +] + +[[package]] +name = "hyperframe" +version = "6.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/02/e7/94f8232d4a74cc99514c13a9f995811485a6903d48e5d952771ef6322e30/hyperframe-6.1.0.tar.gz", hash = "sha256:f630908a00854a7adeabd6382b43923a4c4cd4b821fcb527e6ab9e15382a3b08", size = 26566, upload-time = "2025-01-22T21:41:49.302Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/48/30/47d0bf6072f7252e6521f3447ccfa40b421b6824517f82854703d0f5a98b/hyperframe-6.1.0-py3-none-any.whl", hash = "sha256:b03380493a519fce58ea5af42e4a42317bf9bd425596f7a0835ffce80f1a42e5", size = 13007, upload-time = "2025-01-22T21:41:47.295Z" }, +] + +[[package]] +name = "identify" +version = "2.6.19" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/52/63/51723b5f116cc04b061cb6f5a561790abf249d25931d515cd375e063e0f4/identify-2.6.19.tar.gz", hash = "sha256:6be5020c38fcb07da56c53733538a3081ea5aa70d36a156f83044bfbf9173842", size = 99567, upload-time = "2026-04-17T18:39:50.265Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/94/84/d9273cd09688070a6523c4aee4663a8538721b2b755c4962aafae0011e72/identify-2.6.19-py2.py3-none-any.whl", hash = "sha256:20e6a87f786f768c092a721ad107fc9df0eb89347be9396cadf3f4abbd1fb78a", size = 99397, upload-time = "2026-04-17T18:39:49.221Z" }, +] + +[[package]] +name = "idna" +version = "3.18" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/cd/63/9496c57188a2ee585e0f1db071d75089a11e98aa86eb99d9d7618fc1edce/idna-3.18.tar.gz", hash = "sha256:ffb385a7e039654cef1ab9ef32c6fafe283c0c0467bba1d9029738ce4a14a848", size = 196711, upload-time = "2026-06-02T14:34:07.794Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1e/5e/d4e9f1a599fb8e573b7b87160658329fbf28d19eac2718f51fc3def3aa5a/idna-3.18-py3-none-any.whl", hash = "sha256:7f952cbe720b688055e3f87de14f5c3e5fdaa8bc3928985c4077ca689de849a2", size = 65455, upload-time = "2026-06-02T14:34:06.319Z" }, +] + +[[package]] +name = "importlib-metadata" +version = "8.9.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "zipp" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/e7/72/c600ae4f68c28fc19f9c31b9403053e5dbb8cace2e6842c7b7c3e4d42fe9/importlib_metadata-8.9.0.tar.gz", hash = "sha256:58850626cef4bd2df100378b0f2aea9724a7b92f10770d547725b047078f99ee", size = 56140, upload-time = "2026-03-20T16:56:26.362Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7d/f9/97f2ca8bb3ec6e4b1d64f983ebe98b9a192faddff67fac3d6303a537e670/importlib_metadata-8.9.0-py3-none-any.whl", hash = "sha256:e0f761b6ea91ced3b0844c14c9d955224d538105921f8e6754c00f6ca79fba7f", size = 27220, upload-time = "2026-03-20T16:56:25.07Z" }, +] + +[[package]] +name = "jinja2" +version = "3.1.6" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markupsafe" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/df/bf/f7da0350254c0ed7c72f3e33cef02e048281fec7ecec5f032d4aac52226b/jinja2-3.1.6.tar.gz", hash = "sha256:0137fb05990d35f1275a587e9aee6d56da821fc83491a0fb838183be43f66d6d", size = 245115, upload-time = "2025-03-05T20:05:02.478Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/62/a1/3d680cbfd5f4b8f15abc1d571870c5fc3e594bb582bc3b64ea099db13e56/jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67", size = 134899, upload-time = "2025-03-05T20:05:00.369Z" }, +] + +[[package]] +name = "jiter" +version = "0.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/1d/1f/10936e16d8860c70698a1aa939a46aa0224813b782bce4e000e637da0b2d/jiter-0.16.0.tar.gz", hash = "sha256:7b24c3492c5f4f84a37946ad9cf504910cf6a782d6a4e0689b6673c5894b4a1c", size = 176431, upload-time = "2026-06-29T13:05:13.657Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/83/2b/52ace16ed031354f0539749a49e4bf33797d82bea5137910835fa4b09793/jiter-0.16.0-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:67c3bc1760f8c99d805dcab4e644027142a53b1d5d861f18780ebdbd5d40b72a", size = 306943, upload-time = "2026-06-29T13:03:14.035Z" }, + { url = "https://files.pythonhosted.org/packages/94/2e/34957c2c1b661c252ba9bcc60ae0bddc27e0f7202c6073326a13c5390eec/jiter-0.16.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:5af7780e4a26bd7d0d989592bf9ef12ebf806b74ab709223ecca37c749872ea9", size = 307779, upload-time = "2026-06-29T13:03:15.418Z" }, + { url = "https://files.pythonhosted.org/packages/88/6c/59bd309cab4460c54cf1079f3eb7fe7af6a4c895c5c957a53378693bad2b/jiter-0.16.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d5bf78d0e05e45cfdd66558893938d59afe3d1b1a824a202039b20e607d25a72", size = 335826, upload-time = "2026-06-29T13:03:17.11Z" }, + { url = "https://files.pythonhosted.org/packages/3b/8c/f5ef7b65f0df47afa16596969defb281ebb86e96df346d62be6fd853d620/jiter-0.16.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:f4444a83f946605990c98f625cdd3d2725bfb818158760c5748c653170a20e0e", size = 362573, upload-time = "2026-06-29T13:03:18.781Z" }, + { url = "https://files.pythonhosted.org/packages/2b/0b/ace4354da061ee38844a0c27dc2c21eecd27aea119e8da324bea987522d0/jiter-0.16.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:3a23f0e4f957e1be65752d2dfac9a5a06b1917af8dc85deb639c3b9d02e31290", size = 457979, upload-time = "2026-06-29T13:03:20.293Z" }, + { url = "https://files.pythonhosted.org/packages/55/40/c0253d3772eb9dcd8e6606ee9b2d53ec8e5b814589c47f140aa585f21eaa/jiter-0.16.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:c22a488f7b9218e245a0025a9ba6b100e2e54700831cf4cf16833a27fba3ad01", size = 372302, upload-time = "2026-06-29T13:03:21.739Z" }, + { url = "https://files.pythonhosted.org/packages/a8/d2/4839422241aa12860ce597b20068727094ba0bc480723c74924ca5bad483/jiter-0.16.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:46add52f4ad47a08bfb1219f3e673da972191489a33016edefdb5ea55bfa8c48", size = 343805, upload-time = "2026-06-29T13:03:23.384Z" }, + { url = "https://files.pythonhosted.org/packages/e2/59/e196888a05befdda7dbe299b722d56f2f6eec65402bc34c0a3306d595feb/jiter-0.16.0-cp312-cp312-manylinux_2_31_riscv64.whl", hash = "sha256:9c8a956fd72c2cf1e730d01ea080341f13aa0a97a4a33b51abebe725b7ae9ca9", size = 351107, upload-time = "2026-06-29T13:03:24.815Z" }, + { url = "https://files.pythonhosted.org/packages/ec/74/4cd9e0fca65232136400354b630fbfcd2de634e22ccbb96567725981b548/jiter-0.16.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:561926e0573ffe4a32498420a76d64b16c513e1ab413b9d28158a8764ac701e5", size = 388441, upload-time = "2026-06-29T13:03:26.266Z" }, + { url = "https://files.pythonhosted.org/packages/d9/8c/554691e48bc711299c0a293dd8a6179e24b2d66a54dc295421fcf64569c0/jiter-0.16.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:44d019fa8cdaf89bf29c71b39e3712143fdd0ac76725c6ef954f9957a5ea8730", size = 516354, upload-time = "2026-06-29T13:03:28.02Z" }, + { url = "https://files.pythonhosted.org/packages/a4/cb/01e9d69dc2cc6759d4f91e230b34489c4fdb2518992650633f9e20bece89/jiter-0.16.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:0df91907609837f33341b8e6fe73b95991fdaa57caf1a0fbd343dffe826f386f", size = 547880, upload-time = "2026-06-29T13:03:29.534Z" }, + { url = "https://files.pythonhosted.org/packages/79/70/2953195f1c6ad00f49fa67e13df7e60acb3dd4f387101bc15abccddd905e/jiter-0.16.0-cp312-cp312-win32.whl", hash = "sha256:51d7b836acb0108d7c77df1742332cac2a1fa04a74d6dacec46e7091f0e91274", size = 203473, upload-time = "2026-06-29T13:03:31.025Z" }, + { url = "https://files.pythonhosted.org/packages/2d/05/2909a8b10699a4d560f8c502b6b2c5f3991b682b1922c1eedda242b225bd/jiter-0.16.0-cp312-cp312-win_amd64.whl", hash = "sha256:1878349266f8ee36ecb1375cc5ba2f115f35fd9f0a1a4119e725e379126647f7", size = 196905, upload-time = "2026-06-29T13:03:32.472Z" }, + { url = "https://files.pythonhosted.org/packages/e9/a9/6b82bb1c8d7790d602489b967b982a909e5d092875a6c2ade96444c8dfc5/jiter-0.16.0-cp312-cp312-win_arm64.whl", hash = "sha256:2ed5738ae4af18271a51a528b8811b0cbfa4a1858de9d83359e4169855d6a331", size = 190618, upload-time = "2026-06-29T13:03:34.672Z" }, + { url = "https://files.pythonhosted.org/packages/98/ab/664fd8c4be028b2bedd3d2ff08769c4ede23d0dbc87a77c62384a0515b5d/jiter-0.16.0-graalpy312-graalpy250_312_native-macosx_10_12_x86_64.whl", hash = "sha256:f17d61a28b4b3e0e3e2ba98490c70501403b4d196f78732439160e7fd3678127", size = 303106, upload-time = "2026-06-29T13:05:07.118Z" }, + { url = "https://files.pythonhosted.org/packages/1a/07/421f1d5b65493a76e16027b848aba6a7d28073ae75944fa4289cc914d39f/jiter-0.16.0-graalpy312-graalpy250_312_native-macosx_11_0_arm64.whl", hash = "sha256:96e38eea538c8ddf853a35727c7be0741c76c13f04148ac5c116222f50ece3b3", size = 304658, upload-time = "2026-06-29T13:05:08.708Z" }, + { url = "https://files.pythonhosted.org/packages/0a/db/bba1155f01a01c3c37a89425d571da751bbedf5c54247b831a04cb971798/jiter-0.16.0-graalpy312-graalpy250_312_native-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d284fb8d94d5855d60c44fefcab4bf966f1da6fada73992b01f6f0c9bc0c6702", size = 339719, upload-time = "2026-06-29T13:05:10.41Z" }, + { url = "https://files.pythonhosted.org/packages/78/f7/18a1afcd64f35314b68c1f23afcd9994d0bc13e65cc77517afff4e83986d/jiter-0.16.0-graalpy312-graalpy250_312_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:64d613743df53199b1aa256a7d328340da6d7078aac7705a7db9d7a791e9cfd2", size = 343885, upload-time = "2026-06-29T13:05:12.087Z" }, +] + +[[package]] +name = "jsonschema" +version = "4.26.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "attrs" }, + { name = "jsonschema-specifications" }, + { name = "referencing" }, + { name = "rpds-py" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b3/fc/e067678238fa451312d4c62bf6e6cf5ec56375422aee02f9cb5f909b3047/jsonschema-4.26.0.tar.gz", hash = "sha256:0c26707e2efad8aa1bfc5b7ce170f3fccc2e4918ff85989ba9ffa9facb2be326", size = 366583, upload-time = "2026-01-07T13:41:07.246Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/69/90/f63fb5873511e014207a475e2bb4e8b2e570d655b00ac19a9a0ca0a385ee/jsonschema-4.26.0-py3-none-any.whl", hash = "sha256:d489f15263b8d200f8387e64b4c3a75f06629559fb73deb8fdfb525f2dab50ce", size = 90630, upload-time = "2026-01-07T13:41:05.306Z" }, +] + +[[package]] +name = "jsonschema-specifications" +version = "2025.9.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "referencing" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/19/74/a633ee74eb36c44aa6d1095e7cc5569bebf04342ee146178e2d36600708b/jsonschema_specifications-2025.9.1.tar.gz", hash = "sha256:b540987f239e745613c7a9176f3edb72b832a4ac465cf02712288397832b5e8d", size = 32855, upload-time = "2025-09-08T01:34:59.186Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/41/45/1a4ed80516f02155c51f51e8cedb3c1902296743db0bbc66608a0db2814f/jsonschema_specifications-2025.9.1-py3-none-any.whl", hash = "sha256:98802fee3a11ee76ecaca44429fda8a41bff98b00a0f2838151b113f210cc6fe", size = 18437, upload-time = "2025-09-08T01:34:57.871Z" }, +] + +[[package]] +name = "linkify-it-py" +version = "2.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "uc-micro-py" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/2e/c9/06ea13676ef354f0af6169587ae292d3e2406e212876a413bf9eece4eb23/linkify_it_py-2.1.0.tar.gz", hash = "sha256:43360231720999c10e9328dc3691160e27a718e280673d444c38d7d3aaa3b98b", size = 29158, upload-time = "2026-03-01T07:48:47.683Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b4/de/88b3be5c31b22333b3ca2f6ff1de4e863d8fe45aaea7485f591970ec1d3e/linkify_it_py-2.1.0-py3-none-any.whl", hash = "sha256:0d252c1594ecba2ecedc444053db5d3a9b7ec1b0dd929c8f1d74dce89f86c05e", size = 19878, upload-time = "2026-03-01T07:48:46.098Z" }, +] + +[[package]] +name = "litellm" +version = "1.91.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "aiohttp" }, + { name = "click" }, + { name = "fastuuid" }, + { name = "httpx" }, + { name = "importlib-metadata" }, + { name = "jinja2" }, + { name = "jsonschema" }, + { name = "openai" }, + { name = "pydantic" }, + { name = "python-dotenv" }, + { name = "tiktoken" }, + { name = "tokenizers" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/86/1e/90cfeada42170986a290feebaca0a90923b3c310cddeb0f8690abd239ad4/litellm-1.91.0.tar.gz", hash = "sha256:4fd469fe7356ba8fcc86f4efdf332e3426b760962ab12331fdaf1a01aeec065f", size = 14872290, upload-time = "2026-07-04T19:18:28.466Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a4/15/81fc2d162513803fe08b359c2e2a98f7a33195034fb94b005e263e272c6b/litellm-1.91.0-py3-none-any.whl", hash = "sha256:c3eb52dd2c6a5779e9efd67350f3640f9055d9c05d4b572fc1dd01a217e562ad", size = 16669331, upload-time = "2026-07-04T19:18:25.203Z" }, +] + +[[package]] +name = "markdown-it-py" +version = "4.2.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "mdurl" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/06/ff/7841249c247aa650a76b9ee4bbaeae59370dc8bfd2f6c01f3630c35eb134/markdown_it_py-4.2.0.tar.gz", hash = "sha256:04a21681d6fbb623de53f6f364d352309d4094dd4194040a10fd51833e418d49", size = 82454, upload-time = "2026-05-07T12:08:28.36Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b3/81/4da04ced5a082363ecfa159c010d200ecbd959ae410c10c0264a38cac0f5/markdown_it_py-4.2.0-py3-none-any.whl", hash = "sha256:9f7ebbcd14fe59494226453aed97c1070d83f8d24b6fc3a3bcf9a38092641c4a", size = 91687, upload-time = "2026-05-07T12:08:27.182Z" }, +] + +[package.optional-dependencies] +linkify = [ + { name = "linkify-it-py" }, +] + +[[package]] +name = "markupsafe" +version = "3.0.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7e/99/7690b6d4034fffd95959cbe0c02de8deb3098cc577c67bb6a24fe5d7caa7/markupsafe-3.0.3.tar.gz", hash = "sha256:722695808f4b6457b320fdc131280796bdceb04ab50fe1795cd540799ebe1698", size = 80313, upload-time = "2025-09-27T18:37:40.426Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5a/72/147da192e38635ada20e0a2e1a51cf8823d2119ce8883f7053879c2199b5/markupsafe-3.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:d53197da72cc091b024dd97249dfc7794d6a56530370992a5e1a08983ad9230e", size = 11615, upload-time = "2025-09-27T18:36:30.854Z" }, + { url = "https://files.pythonhosted.org/packages/9a/81/7e4e08678a1f98521201c3079f77db69fb552acd56067661f8c2f534a718/markupsafe-3.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:1872df69a4de6aead3491198eaf13810b565bdbeec3ae2dc8780f14458ec73ce", size = 12020, upload-time = "2025-09-27T18:36:31.971Z" }, + { url = "https://files.pythonhosted.org/packages/1e/2c/799f4742efc39633a1b54a92eec4082e4f815314869865d876824c257c1e/markupsafe-3.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3a7e8ae81ae39e62a41ec302f972ba6ae23a5c5396c8e60113e9066ef893da0d", size = 24332, upload-time = "2025-09-27T18:36:32.813Z" }, + { url = "https://files.pythonhosted.org/packages/3c/2e/8d0c2ab90a8c1d9a24f0399058ab8519a3279d1bd4289511d74e909f060e/markupsafe-3.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d6dd0be5b5b189d31db7cda48b91d7e0a9795f31430b7f271219ab30f1d3ac9d", size = 22947, upload-time = "2025-09-27T18:36:33.86Z" }, + { url = "https://files.pythonhosted.org/packages/2c/54/887f3092a85238093a0b2154bd629c89444f395618842e8b0c41783898ea/markupsafe-3.0.3-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:94c6f0bb423f739146aec64595853541634bde58b2135f27f61c1ffd1cd4d16a", size = 21962, upload-time = "2025-09-27T18:36:35.099Z" }, + { url = "https://files.pythonhosted.org/packages/c9/2f/336b8c7b6f4a4d95e91119dc8521402461b74a485558d8f238a68312f11c/markupsafe-3.0.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:be8813b57049a7dc738189df53d69395eba14fb99345e0a5994914a3864c8a4b", size = 23760, upload-time = "2025-09-27T18:36:36.001Z" }, + { url = "https://files.pythonhosted.org/packages/32/43/67935f2b7e4982ffb50a4d169b724d74b62a3964bc1a9a527f5ac4f1ee2b/markupsafe-3.0.3-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:83891d0e9fb81a825d9a6d61e3f07550ca70a076484292a70fde82c4b807286f", size = 21529, upload-time = "2025-09-27T18:36:36.906Z" }, + { url = "https://files.pythonhosted.org/packages/89/e0/4486f11e51bbba8b0c041098859e869e304d1c261e59244baa3d295d47b7/markupsafe-3.0.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:77f0643abe7495da77fb436f50f8dab76dbc6e5fd25d39589a0f1fe6548bfa2b", size = 23015, upload-time = "2025-09-27T18:36:37.868Z" }, + { url = "https://files.pythonhosted.org/packages/2f/e1/78ee7a023dac597a5825441ebd17170785a9dab23de95d2c7508ade94e0e/markupsafe-3.0.3-cp312-cp312-win32.whl", hash = "sha256:d88b440e37a16e651bda4c7c2b930eb586fd15ca7406cb39e211fcff3bf3017d", size = 14540, upload-time = "2025-09-27T18:36:38.761Z" }, + { url = "https://files.pythonhosted.org/packages/aa/5b/bec5aa9bbbb2c946ca2733ef9c4ca91c91b6a24580193e891b5f7dbe8e1e/markupsafe-3.0.3-cp312-cp312-win_amd64.whl", hash = "sha256:26a5784ded40c9e318cfc2bdb30fe164bdb8665ded9cd64d500a34fb42067b1c", size = 15105, upload-time = "2025-09-27T18:36:39.701Z" }, + { url = "https://files.pythonhosted.org/packages/e5/f1/216fc1bbfd74011693a4fd837e7026152e89c4bcf3e77b6692fba9923123/markupsafe-3.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:35add3b638a5d900e807944a078b51922212fb3dedb01633a8defc4b01a3c85f", size = 13906, upload-time = "2025-09-27T18:36:40.689Z" }, +] + +[[package]] +name = "mdit-py-plugins" +version = "0.6.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markdown-it-py" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/59/fc/f8d0863f8862f25602c0404d75568e89fb6b4109804645e5cdfb1be5cf56/mdit_py_plugins-0.6.1.tar.gz", hash = "sha256:a2bca0f039f39dbd35fb74ae1b5f998608c437463371f0ff7f49a19a17a114d0", size = 56114, upload-time = "2026-05-13T09:03:38.91Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a5/69/6da5581c6a7fede7dc261bf4e67d6adca4196f176b43288b55b3db395b6e/mdit_py_plugins-0.6.1-py3-none-any.whl", hash = "sha256:214c82fb2ac524472ab6a5bcab1de80f73b50443e187f401bfd77efbc7c6481d", size = 66663, upload-time = "2026-05-13T09:03:37.76Z" }, +] + +[[package]] +name = "mdurl" +version = "0.1.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d6/54/cfe61301667036ec958cb99bd3efefba235e65cdeb9c84d24a8293ba1d90/mdurl-0.1.2.tar.gz", hash = "sha256:bb413d29f5eea38f31dd4754dd7377d4465116fb207585f97bf925588687c1ba", size = 8729, upload-time = "2022-08-14T12:40:10.846Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl", hash = "sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8", size = 9979, upload-time = "2022-08-14T12:40:09.779Z" }, +] + +[[package]] +name = "mini-swe-agent" +version = "2.3.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "datasets" }, + { name = "jinja2" }, + { name = "litellm" }, + { name = "openai" }, + { name = "platformdirs" }, + { name = "prompt-toolkit" }, + { name = "pydantic" }, + { name = "python-dotenv" }, + { name = "pyyaml" }, + { name = "requests" }, + { name = "rich" }, + { name = "tenacity" }, + { name = "textual" }, + { name = "typer" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/2e/49/cfc66c0ec44d829e1d01b9e756f54294854a51362f4d54c68b81c7b49928/mini_swe_agent-2.3.0.tar.gz", hash = "sha256:8aae4f2a603a1f971dd183a5ca5157016e130d6f67f0ff35ad148d014673d41f", size = 63780, upload-time = "2026-05-21T14:37:59.703Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a6/f5/941c20e2152cc3bd0a85773ab62640c1d0a71972bff1964e3d059eceddf9/mini_swe_agent-2.3.0-py3-none-any.whl", hash = "sha256:27f68d784d3a22768fc389f31a451bb55e5d00fa46e1a27032d029da35878988", size = 110649, upload-time = "2026-05-21T14:37:58.548Z" }, +] + +[[package]] +name = "modal" +version = "1.5.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "aiohttp" }, + { name = "cbor2" }, + { name = "certifi" }, + { name = "click" }, + { name = "grpclib" }, + { name = "protobuf" }, + { name = "rich" }, + { name = "synchronicity" }, + { name = "toml" }, + { name = "types-certifi" }, + { name = "types-toml" }, + { name = "typing-extensions" }, + { name = "watchfiles" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b8/e6/44f609084472d16a3acf7748c184e23d3009944eed5860e855aaafc61065/modal-1.5.2.tar.gz", hash = "sha256:ac0d2983ecccacb80c54385906404d75f37a3751f1374c750974cd7853c0008a", size = 818040, upload-time = "2026-07-10T18:12:55.164Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e3/fe/7e284d889eb3ea2c307e48828eccdd9a984beb2e096bce5577ef796592e9/modal-1.5.2-py3-none-any.whl", hash = "sha256:7508a44f8742c6f26be4d28da71f46092ef3bcc07d0f27cbd0ea8c9d6d9fc873", size = 932258, upload-time = "2026-07-10T18:12:52.219Z" }, +] + +[[package]] +name = "msgspec" +version = "0.20.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ea/9c/bfbd12955a49180cbd234c5d29ec6f74fe641698f0cd9df154a854fc8a15/msgspec-0.20.0.tar.gz", hash = "sha256:692349e588fde322875f8d3025ac01689fead5901e7fb18d6870a44519d62a29", size = 317862, upload-time = "2025-11-24T03:56:28.934Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d9/6f/1e25eee957e58e3afb2a44b94fa95e06cebc4c236193ed0de3012fff1e19/msgspec-0.20.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:2aba22e2e302e9231e85edc24f27ba1f524d43c223ef5765bd8624c7df9ec0a5", size = 196391, upload-time = "2025-11-24T03:55:32.677Z" }, + { url = "https://files.pythonhosted.org/packages/7f/ee/af51d090ada641d4b264992a486435ba3ef5b5634bc27e6eb002f71cef7d/msgspec-0.20.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:716284f898ab2547fedd72a93bb940375de9fbfe77538f05779632dc34afdfde", size = 188644, upload-time = "2025-11-24T03:55:33.934Z" }, + { url = "https://files.pythonhosted.org/packages/49/d6/9709ee093b7742362c2934bfb1bbe791a1e09bed3ea5d8a18ce552fbfd73/msgspec-0.20.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:558ed73315efa51b1538fa8f1d3b22c8c5ff6d9a2a62eff87d25829b94fc5054", size = 218852, upload-time = "2025-11-24T03:55:35.575Z" }, + { url = "https://files.pythonhosted.org/packages/5c/a2/488517a43ccf5a4b6b6eca6dd4ede0bd82b043d1539dd6bb908a19f8efd3/msgspec-0.20.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:509ac1362a1d53aa66798c9b9fd76872d7faa30fcf89b2fba3bcbfd559d56eb0", size = 224937, upload-time = "2025-11-24T03:55:36.859Z" }, + { url = "https://files.pythonhosted.org/packages/d5/e8/49b832808aa23b85d4f090d1d2e48a4e3834871415031ed7c5fe48723156/msgspec-0.20.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:1353c2c93423602e7dea1aa4c92f3391fdfc25ff40e0bacf81d34dbc68adb870", size = 222858, upload-time = "2025-11-24T03:55:38.187Z" }, + { url = "https://files.pythonhosted.org/packages/9f/56/1dc2fa53685dca9c3f243a6cbecd34e856858354e455b77f47ebd76cf5bf/msgspec-0.20.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:cb33b5eb5adb3c33d749684471c6a165468395d7aa02d8867c15103b81e1da3e", size = 227248, upload-time = "2025-11-24T03:55:39.496Z" }, + { url = "https://files.pythonhosted.org/packages/5a/51/aba940212c23b32eedce752896205912c2668472ed5b205fc33da28a6509/msgspec-0.20.0-cp312-cp312-win_amd64.whl", hash = "sha256:fb1d934e435dd3a2b8cf4bbf47a8757100b4a1cfdc2afdf227541199885cdacb", size = 190024, upload-time = "2025-11-24T03:55:40.829Z" }, + { url = "https://files.pythonhosted.org/packages/41/ad/3b9f259d94f183daa9764fef33fdc7010f7ecffc29af977044fa47440a83/msgspec-0.20.0-cp312-cp312-win_arm64.whl", hash = "sha256:00648b1e19cf01b2be45444ba9dc961bd4c056ffb15706651e64e5d6ec6197b7", size = 175390, upload-time = "2025-11-24T03:55:42.05Z" }, +] + +[[package]] +name = "multidict" +version = "6.7.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/1a/c2/c2d94cbe6ac1753f3fc980da97b3d930efe1da3af3c9f5125354436c073d/multidict-6.7.1.tar.gz", hash = "sha256:ec6652a1bee61c53a3e5776b6049172c53b6aaba34f18c9ad04f82712bac623d", size = 102010, upload-time = "2026-01-26T02:46:45.979Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/8d/9c/f20e0e2cf80e4b2e4b1c365bf5fe104ee633c751a724246262db8f1a0b13/multidict-6.7.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:a90f75c956e32891a4eda3639ce6dd86e87105271f43d43442a3aedf3cddf172", size = 76893, upload-time = "2026-01-26T02:43:52.754Z" }, + { url = "https://files.pythonhosted.org/packages/fe/cf/18ef143a81610136d3da8193da9d80bfe1cb548a1e2d1c775f26b23d024a/multidict-6.7.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:3fccb473e87eaa1382689053e4a4618e7ba7b9b9b8d6adf2027ee474597128cd", size = 45456, upload-time = "2026-01-26T02:43:53.893Z" }, + { url = "https://files.pythonhosted.org/packages/a9/65/1caac9d4cd32e8433908683446eebc953e82d22b03d10d41a5f0fefe991b/multidict-6.7.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:b0fa96985700739c4c7853a43c0b3e169360d6855780021bfc6d0f1ce7c123e7", size = 43872, upload-time = "2026-01-26T02:43:55.041Z" }, + { url = "https://files.pythonhosted.org/packages/cf/3b/d6bd75dc4f3ff7c73766e04e705b00ed6dbbaccf670d9e05a12b006f5a21/multidict-6.7.1-cp312-cp312-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:cb2a55f408c3043e42b40cc8eecd575afa27b7e0b956dfb190de0f8499a57a53", size = 251018, upload-time = "2026-01-26T02:43:56.198Z" }, + { url = "https://files.pythonhosted.org/packages/fd/80/c959c5933adedb9ac15152e4067c702a808ea183a8b64cf8f31af8ad3155/multidict-6.7.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:eb0ce7b2a32d09892b3dd6cc44877a0d02a33241fafca5f25c8b6b62374f8b75", size = 258883, upload-time = "2026-01-26T02:43:57.499Z" }, + { url = "https://files.pythonhosted.org/packages/86/85/7ed40adafea3d4f1c8b916e3b5cc3a8e07dfcdcb9cd72800f4ed3ca1b387/multidict-6.7.1-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:c3a32d23520ee37bf327d1e1a656fec76a2edd5c038bf43eddfa0572ec49c60b", size = 242413, upload-time = "2026-01-26T02:43:58.755Z" }, + { url = "https://files.pythonhosted.org/packages/d2/57/b8565ff533e48595503c785f8361ff9a4fde4d67de25c207cd0ba3befd03/multidict-6.7.1-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9c90fed18bffc0189ba814749fdcc102b536e83a9f738a9003e569acd540a733", size = 268404, upload-time = "2026-01-26T02:44:00.216Z" }, + { url = "https://files.pythonhosted.org/packages/e0/50/9810c5c29350f7258180dfdcb2e52783a0632862eb334c4896ac717cebcb/multidict-6.7.1-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:da62917e6076f512daccfbbde27f46fed1c98fee202f0559adec8ee0de67f71a", size = 269456, upload-time = "2026-01-26T02:44:02.202Z" }, + { url = "https://files.pythonhosted.org/packages/f3/8d/5e5be3ced1d12966fefb5c4ea3b2a5b480afcea36406559442c6e31d4a48/multidict-6.7.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bfde23ef6ed9db7eaee6c37dcec08524cb43903c60b285b172b6c094711b3961", size = 256322, upload-time = "2026-01-26T02:44:03.56Z" }, + { url = "https://files.pythonhosted.org/packages/31/6e/d8a26d81ac166a5592782d208dd90dfdc0a7a218adaa52b45a672b46c122/multidict-6.7.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:3758692429e4e32f1ba0df23219cd0b4fc0a52f476726fff9337d1a57676a582", size = 253955, upload-time = "2026-01-26T02:44:04.845Z" }, + { url = "https://files.pythonhosted.org/packages/59/4c/7c672c8aad41534ba619bcd4ade7a0dc87ed6b8b5c06149b85d3dd03f0cd/multidict-6.7.1-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:398c1478926eca669f2fd6a5856b6de9c0acf23a2cb59a14c0ba5844fa38077e", size = 251254, upload-time = "2026-01-26T02:44:06.133Z" }, + { url = "https://files.pythonhosted.org/packages/7b/bd/84c24de512cbafbdbc39439f74e967f19570ce7924e3007174a29c348916/multidict-6.7.1-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:c102791b1c4f3ab36ce4101154549105a53dc828f016356b3e3bcae2e3a039d3", size = 252059, upload-time = "2026-01-26T02:44:07.518Z" }, + { url = "https://files.pythonhosted.org/packages/fa/ba/f5449385510825b73d01c2d4087bf6d2fccc20a2d42ac34df93191d3dd03/multidict-6.7.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:a088b62bd733e2ad12c50dad01b7d0166c30287c166e137433d3b410add807a6", size = 263588, upload-time = "2026-01-26T02:44:09.382Z" }, + { url = "https://files.pythonhosted.org/packages/d7/11/afc7c677f68f75c84a69fe37184f0f82fce13ce4b92f49f3db280b7e92b3/multidict-6.7.1-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:3d51ff4785d58d3f6c91bdbffcb5e1f7ddfda557727043aa20d20ec4f65e324a", size = 259642, upload-time = "2026-01-26T02:44:10.73Z" }, + { url = "https://files.pythonhosted.org/packages/2b/17/ebb9644da78c4ab36403739e0e6e0e30ebb135b9caf3440825001a0bddcb/multidict-6.7.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:fc5907494fccf3e7d3f94f95c91d6336b092b5fc83811720fae5e2765890dfba", size = 251377, upload-time = "2026-01-26T02:44:12.042Z" }, + { url = "https://files.pythonhosted.org/packages/ca/a4/840f5b97339e27846c46307f2530a2805d9d537d8b8bd416af031cad7fa0/multidict-6.7.1-cp312-cp312-win32.whl", hash = "sha256:28ca5ce2fd9716631133d0e9a9b9a745ad7f60bac2bccafb56aa380fc0b6c511", size = 41887, upload-time = "2026-01-26T02:44:14.245Z" }, + { url = "https://files.pythonhosted.org/packages/80/31/0b2517913687895f5904325c2069d6a3b78f66cc641a86a2baf75a05dcbb/multidict-6.7.1-cp312-cp312-win_amd64.whl", hash = "sha256:fcee94dfbd638784645b066074b338bc9cc155d4b4bffa4adce1615c5a426c19", size = 46053, upload-time = "2026-01-26T02:44:15.371Z" }, + { url = "https://files.pythonhosted.org/packages/0c/5b/aba28e4ee4006ae4c7df8d327d31025d760ffa992ea23812a601d226e682/multidict-6.7.1-cp312-cp312-win_arm64.whl", hash = "sha256:ba0a9fb644d0c1a2194cf7ffb043bd852cea63a57f66fbd33959f7dae18517bf", size = 43307, upload-time = "2026-01-26T02:44:16.852Z" }, + { url = "https://files.pythonhosted.org/packages/81/08/7036c080d7117f28a4af526d794aab6a84463126db031b007717c1a6676e/multidict-6.7.1-py3-none-any.whl", hash = "sha256:55d97cc6dae627efa6a6e548885712d4864b81110ac76fa4e534c03819fa4a56", size = 12319, upload-time = "2026-01-26T02:46:44.004Z" }, +] + +[[package]] +name = "multiprocess" +version = "0.70.19" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "dill" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a2/f2/e783ac7f2aeeed14e9e12801f22529cc7e6b7ab80928d6dcce4e9f00922d/multiprocess-0.70.19.tar.gz", hash = "sha256:952021e0e6c55a4a9fe4cd787895b86e239a40e76802a789d6305398d3975897", size = 2079989, upload-time = "2026-01-19T06:47:39.744Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e3/45/8004d1e6b9185c1a444d6b55ac5682acf9d98035e54386d967366035a03a/multiprocess-0.70.19-py310-none-any.whl", hash = "sha256:97404393419dcb2a8385910864eedf47a3cadf82c66345b44f036420eb0b5d87", size = 134948, upload-time = "2026-01-19T06:47:32.325Z" }, + { url = "https://files.pythonhosted.org/packages/86/c2/dec9722dc3474c164a0b6bcd9a7ed7da542c98af8cabce05374abab35edd/multiprocess-0.70.19-py311-none-any.whl", hash = "sha256:928851ae7973aea4ce0eaf330bbdafb2e01398a91518d5c8818802845564f45c", size = 144457, upload-time = "2026-01-19T06:47:33.711Z" }, + { url = "https://files.pythonhosted.org/packages/71/70/38998b950a97ea279e6bd657575d22d1a2047256caf707d9a10fbce4f065/multiprocess-0.70.19-py312-none-any.whl", hash = "sha256:3a56c0e85dd5025161bac5ce138dcac1e49174c7d8e74596537e729fd5c53c28", size = 150281, upload-time = "2026-01-19T06:47:35.037Z" }, + { url = "https://files.pythonhosted.org/packages/7e/82/69e539c4c2027f1e1697e09aaa2449243085a0edf81ae2c6341e84d769b6/multiprocess-0.70.19-py39-none-any.whl", hash = "sha256:0d4b4397ed669d371c81dcd1ef33fd384a44d6c3de1bd0ca7ac06d837720d3c5", size = 133477, upload-time = "2026-01-19T06:47:38.619Z" }, +] + +[[package]] +name = "nodeenv" +version = "1.10.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/24/bf/d1bda4f6168e0b2e9e5958945e01910052158313224ada5ce1fb2e1113b8/nodeenv-1.10.0.tar.gz", hash = "sha256:996c191ad80897d076bdfba80a41994c2b47c68e224c542b48feba42ba00f8bb", size = 55611, upload-time = "2025-12-20T14:08:54.006Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/88/b2/d0896bdcdc8d28a7fc5717c305f1a861c26e18c05047949fb371034d98bd/nodeenv-1.10.0-py2.py3-none-any.whl", hash = "sha256:5bb13e3eed2923615535339b3c620e76779af4cb4c6a90deccc9e36b274d3827", size = 23438, upload-time = "2025-12-20T14:08:52.782Z" }, +] + +[[package]] +name = "numpy" +version = "2.5.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/22/fd/89965aa4ac08c74998539fcbf24fa3540f3e15237fbeb6bcf9c908f4aade/numpy-2.5.1.tar.gz", hash = "sha256:a48a113e6afea91f5608793bafa7ef2ad481fefbda87ec5069f483de61cb9fa3", size = 20755553, upload-time = "2026-07-04T17:08:00.933Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/62/7b/14687aa674250e5e546f616f486b0d56d3631cd5b2415739141ce40bdcea/numpy-2.5.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:2c889b56fe48b1018f764b0eec8df59ab654e9148aa91faa12596043500de277", size = 16801574, upload-time = "2026-07-04T17:06:12.423Z" }, + { url = "https://files.pythonhosted.org/packages/e1/19/cc5bb2a3f2913d27d6dbb2c78d25921fabaedc6741d4a5a615a11f3c5bf3/numpy-2.5.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:ab451b59c5643c570974c43aef780703ef1d3b4965d2be07afd530615a9358d1", size = 11772250, upload-time = "2026-07-04T17:06:15.726Z" }, + { url = "https://files.pythonhosted.org/packages/42/77/fdf34a71dd30f54979b18603bee915e0aaf825b07afe79acd60b04b691e2/numpy-2.5.1-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:78798bd5b9ad744056af8efa90e3b9ddaa53272a0848a483084a1cc0a13b2dc0", size = 5331516, upload-time = "2026-07-04T17:06:17.913Z" }, + { url = "https://files.pythonhosted.org/packages/ce/e2/eb7efa015b4cce41e2517bf182a7fce0d7d5b9d9ed76a29bfa0f4fe4505c/numpy-2.5.1-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:2ae0ca40bcb22d6ba59c1dfd5446f49940b0f2d821fde133f10dda11f816b84e", size = 6664863, upload-time = "2026-07-04T17:06:20.02Z" }, + { url = "https://files.pythonhosted.org/packages/a9/4b/a2b32dd94ee9ffbeecb28152240042a3949db33b1c834d44090b80e1b3b8/numpy-2.5.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:61ac47e772e6b8ea489e1d2f441a34c5c3ac17327e7ce294cbdf535795ad4e75", size = 15167977, upload-time = "2026-07-04T17:06:21.621Z" }, + { url = "https://files.pythonhosted.org/packages/b8/a9/6e73d68500f80773f65f0654ea932019d6694329a0eb0ed0533de38df376/numpy-2.5.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:59fda5e192b570217ec2580c96f00e9a7e12ef6866a900eb089b62c1a32545ca", size = 16672469, upload-time = "2026-07-04T17:06:24.064Z" }, + { url = "https://files.pythonhosted.org/packages/24/7d/ad3e59015135f5261c95fd4cafeff159c955febd83a99a1d9250c4233815/numpy-2.5.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:f7119ebff1a9829e9f431a4f9d28e703023bb6b9fe7c8f724467dbfc27c94ab3", size = 16527531, upload-time = "2026-07-04T17:06:26.69Z" }, + { url = "https://files.pythonhosted.org/packages/83/d0/a39b2fbcde9cb17a1dac678f254b33a6336298af9df338824c685425d5e8/numpy-2.5.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:e824c2acf8862052246be5a44c15da1777940c60d010dd2aab897824d9c430f9", size = 18431940, upload-time = "2026-07-04T17:06:29.521Z" }, + { url = "https://files.pythonhosted.org/packages/04/12/cff070947791c1ed425ff76413189adbdc2fbe215eba7ce7fa454a03c7f8/numpy-2.5.1-cp312-cp312-win32.whl", hash = "sha256:08d60c810432eb83360958dea0999ac4cfb94531ea8efcbf0b7f277c2068aeb2", size = 6066764, upload-time = "2026-07-04T17:06:32.571Z" }, + { url = "https://files.pythonhosted.org/packages/65/66/53f31807a48a750f9d748da273bc3fcedd12b27ff1f3e373bfec55ef2dc0/numpy-2.5.1-cp312-cp312-win_amd64.whl", hash = "sha256:f7d60026c0bdb1380e83bfa7a0419c4577ee4b9a08880afcb6dadeb74c649fa2", size = 12430966, upload-time = "2026-07-04T17:06:34.926Z" }, + { url = "https://files.pythonhosted.org/packages/2b/2a/d1a88066b1c14186f5d3c0d18c94f17b064511982bab0578d49ee9d43c29/numpy-2.5.1-cp312-cp312-win_arm64.whl", hash = "sha256:17a25e09640602e10bc8de0e6fa2b3fd68eedd84ba6d7842dc8f32f9ab87bd0b", size = 10350488, upload-time = "2026-07-04T17:06:37.785Z" }, +] + +[[package]] +name = "openai" +version = "2.45.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "distro" }, + { name = "httpx" }, + { name = "jiter" }, + { name = "pydantic" }, + { name = "sniffio" }, + { name = "tqdm" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/78/60/d4219875289b11d2c2f7da93c36283da224a2e55865ed865ab64e0ce9217/openai-2.45.0.tar.gz", hash = "sha256:10d34ca9c5643bce775852fddbfc172505cb1d4de1ccd101696c3ecff358765d", size = 1109653, upload-time = "2026-07-09T18:02:44.091Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f1/b0/2291689e3ec4723fbf5bbf3b54afcd7b160f9ddc98ca7aedfd0132af5677/openai-2.45.0-py3-none-any.whl", hash = "sha256:5df105f5f8c9b711fcb9d06d2d3888cebc82506db216484c14a4e53cdf651777", size = 1629470, upload-time = "2026-07-09T18:02:42.21Z" }, +] + +[[package]] +name = "packaging" +version = "26.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d7/f1/e7a6dd94a8d4a5626c03e4e99c87f241ba9e350cd9e6d75123f992427270/packaging-26.2.tar.gz", hash = "sha256:ff452ff5a3e828ce110190feff1178bb1f2ea2281fa2075aadb987c2fb221661", size = 228134, upload-time = "2026-04-24T20:15:23.917Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/df/b2/87e62e8c3e2f4b32e5fe99e0b86d576da1312593b39f47d8ceef365e95ed/packaging-26.2-py3-none-any.whl", hash = "sha256:5fc45236b9446107ff2415ce77c807cee2862cb6fac22b8a73826d0693b0980e", size = 100195, upload-time = "2026-04-24T20:15:22.081Z" }, +] + +[[package]] +name = "pandas" +version = "3.0.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy" }, + { name = "python-dateutil" }, + { name = "tzdata", marker = "sys_platform == 'emscripten' or sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/f8/87/4341c6252d1c47b08768c3d25ac487362bf403f0313ddae4a2a26c9b1b4c/pandas-3.0.3.tar.gz", hash = "sha256:696a4a00a2a2a35d4e5deb3fc946641b96c944f02230e4f76137fe35d806c4fc", size = 4651414, upload-time = "2026-05-11T18:54:29.21Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/24/f1/392f8c5bfc16f66a0d2d41561c01627c228fe7ed2a0d056ef11315042570/pandas-3.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:fed2ff7fd9779120e388e285fc029bd5cf9490cdd2e4166a9ee22c0e49a9ab09", size = 10357846, upload-time = "2026-05-11T18:52:36.143Z" }, + { url = "https://files.pythonhosted.org/packages/cf/3d/b16412745651e855f357e5e66930248688378853a6e2698a214e331fba1f/pandas-3.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:b168fc218fd80a6cbdbdbc1a97ddc7889ed057d7eb45f50d866ceab5f39904c4", size = 9899550, upload-time = "2026-05-11T18:52:38.976Z" }, + { url = "https://files.pythonhosted.org/packages/31/a8/fa2535168fffcedf67f4f6de28d2dd903a747ca7c8ea6989451aaeb3a92f/pandas-3.0.3-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0383c72c75cdcca61a9e116e611143902dbfd08bff356829c2f6d1cf40a9ca8c", size = 10412965, upload-time = "2026-05-11T18:52:41.915Z" }, + { url = "https://files.pythonhosted.org/packages/65/b6/09b01cdbc15224e2850365192d17b7bdebb8bdbd8780ed221fcdf0d9a515/pandas-3.0.3-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6dc0b3fd2169c9157deed50b4d519553a3655c8c6a96027136d654592be973a9", size = 10894600, upload-time = "2026-05-11T18:52:45.02Z" }, + { url = "https://files.pythonhosted.org/packages/c9/a4/2eb28f2fccb4ced4a2c79ab2a5dee9ade1ebf44922ebad6fea158c9f95d4/pandas-3.0.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:7e65d5407dc0b394f509699650e4a2ec01c0514f21850f453fa60f3be79a5dbf", size = 11422824, upload-time = "2026-05-11T18:52:48.058Z" }, + { url = "https://files.pythonhosted.org/packages/f8/45/830bb57f533a4604b355e07edcb8ea18cf88b5f94e5fca92f27052d7c597/pandas-3.0.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:f8894dc474d648fe7b6ff0ca9b0bd73950d19952bc1a6534540762c5d79d305c", size = 11950889, upload-time = "2026-05-11T18:52:50.905Z" }, + { url = "https://files.pythonhosted.org/packages/b9/c5/fc1b368f303087d20e8c9bf3d6ceb186263cfac0ade735cd938538bea839/pandas-3.0.3-cp312-cp312-win_amd64.whl", hash = "sha256:c7be265b62cef88e253a941e4698604973736dcfe242fdb5198f0f7bc473cdcc", size = 9755463, upload-time = "2026-05-11T18:52:53.386Z" }, + { url = "https://files.pythonhosted.org/packages/86/bd/fda8f9705b1b09c6ebe14bfc0fa0e4ec8584d54ea673628f157ff55131af/pandas-3.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:557409bc4178e70ee8d9ddb494798e51ebf6ea59330f6be22c51bab2a7db6c49", size = 9066158, upload-time = "2026-05-11T18:52:56.038Z" }, +] + +[[package]] +name = "platformdirs" +version = "4.10.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d7/47/e4501f49c178ae1d9f4a75073fda4204f52647993f075a9db4d14930e0c5/platformdirs-4.10.0.tar.gz", hash = "sha256:31e761a6a0ca04faf7353ea759bdba55652be214725111e5aac52dfa29d4bef7", size = 31224, upload-time = "2026-05-28T03:32:53.587Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/81/e6/cd9575ac904136b3cbf7aa7ee819ef86eedb7274e46f230e94ea4342e729/platformdirs-4.10.0-py3-none-any.whl", hash = "sha256:fb516cdb12eb0d857d0cd85a7c57cea4d060bee4578d6cf5a14dfdf8cbf8784a", size = 22743, upload-time = "2026-05-28T03:32:52.175Z" }, +] + +[[package]] +name = "pre-commit" +version = "4.6.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "cfgv" }, + { name = "identify" }, + { name = "nodeenv" }, + { name = "pyyaml" }, + { name = "virtualenv" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/8e/22/2de9408ac81acbb8a7d05d4cc064a152ccf33b3d480ebe0cd292153db239/pre_commit-4.6.0.tar.gz", hash = "sha256:718d2208cef53fdc38206e40524a6d4d9576d103eb16f0fec11c875e7716e9d9", size = 198525, upload-time = "2026-04-21T20:31:41.613Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/80/6e/4b28b62ecb6aae56769c34a8ff1d661473ec1e9519e2d5f8b2c150086b26/pre_commit-4.6.0-py2.py3-none-any.whl", hash = "sha256:e2cf246f7299edcabcf15f9b0571fdce06058527f0a06535068a86d38089f29b", size = 226472, upload-time = "2026-04-21T20:31:40.092Z" }, +] + +[[package]] +name = "prompt-toolkit" +version = "3.0.52" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "wcwidth" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a1/96/06e01a7b38dce6fe1db213e061a4602dd6032a8a97ef6c1a862537732421/prompt_toolkit-3.0.52.tar.gz", hash = "sha256:28cde192929c8e7321de85de1ddbe736f1375148b02f2e17edd840042b1be855", size = 434198, upload-time = "2025-08-27T15:24:02.057Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/84/03/0d3ce49e2505ae70cf43bc5bb3033955d2fc9f932163e84dc0779cc47f48/prompt_toolkit-3.0.52-py3-none-any.whl", hash = "sha256:9aac639a3bbd33284347de5ad8d68ecc044b91a762dc39b7c21095fcd6a19955", size = 391431, upload-time = "2025-08-27T15:23:59.498Z" }, +] + +[[package]] +name = "propcache" +version = "0.5.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ec/44/c87281c333769159c50594f22610f77398a47ccbfbbf23074e744e86f87c/propcache-0.5.2.tar.gz", hash = "sha256:01c4fc7480cd0598bb4b57022df55b9ca296da7fc5a8760bd8451a7e63a7d427", size = 50208, upload-time = "2026-05-08T21:02:12.199Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4a/cb/e27bc2b2737a0bb49962b275efa051e8f1c35a936df7d5139b6b658b7dc9/propcache-0.5.2-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:806719138ecd720339a12410fb9614ac9b2b2d3a5fdf8235d56981c36f4039ba", size = 95887, upload-time = "2026-05-08T21:00:11.277Z" }, + { url = "https://files.pythonhosted.org/packages/e6/13/b8ae04c59392f8d11c6cd9fb4011d1dc7c86b81225c770280300e259ffe1/propcache-0.5.2-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:db2b80ea58eab4f86b2beec3cc8b39e8ff9276ac20e96b7cce43c8ae84cd6b5a", size = 54654, upload-time = "2026-05-08T21:00:12.604Z" }, + { url = "https://files.pythonhosted.org/packages/2c/7d/49777a3e20b55863d4794384a38acd460c04157b0a00f8602b0d508b8431/propcache-0.5.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:e5cbfac9f61484f7e9f3597775500cd3ebe8274e9b050c38f9525c77c97520bf", size = 55190, upload-time = "2026-05-08T21:00:13.935Z" }, + { url = "https://files.pythonhosted.org/packages/44/c7/085d0cd63062e84044e3f05797749c3f8e3938ff3aeb0eb2f69d43fafc91/propcache-0.5.2-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5dbc581d2814337da56222fab8dc5f161cd798a434e49bac27930aaef798e144", size = 59995, upload-time = "2026-05-08T21:00:15.526Z" }, + { url = "https://files.pythonhosted.org/packages/9c/42/32cf8e3009e92b2645cf1e944f701e8ea4e924dffde1ee26db860bcbf7e4/propcache-0.5.2-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:857187f381f88c8e2fa2fe56ab94879d011b883d5a2ee5a1b60a8cd2a06846d9", size = 63422, upload-time = "2026-05-08T21:00:16.824Z" }, + { url = "https://files.pythonhosted.org/packages/9e/1b/f112433f99fc979431b87a39ef169e3f8df070d99a72792c56d6937ac48b/propcache-0.5.2-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:178b4a2cdaac1818e2bf1c5a99b94383fa73ea5382e032a48dec07dc5668dc42", size = 64342, upload-time = "2026-05-08T21:00:18.362Z" }, + { url = "https://files.pythonhosted.org/packages/14/15/5574111ae50dd6e879456888c0eadd4c5a869959775854e18e18a6b345f3/propcache-0.5.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6f328175a2cde1f0ff2c4ed8ce968b9dcfb55f3a7153f39e2957ed994da13476", size = 61639, upload-time = "2026-05-08T21:00:19.692Z" }, + { url = "https://files.pythonhosted.org/packages/cc/da/4d775080b1490c0ae604acda868bd71aabe3a89ed16f2aa4339eb8a283e7/propcache-0.5.2-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:5671d09a36b06d0fd4a3da0fccbcae360e9b1570924171a15e9e0997f0249fba", size = 61588, upload-time = "2026-05-08T21:00:21.155Z" }, + { url = "https://files.pythonhosted.org/packages/04/ac/f076982cbe2195ee9cf32de5a1e46951d9fb399fc207f390562dd0fd8fb2/propcache-0.5.2-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:80168e2ebe4d3ec6599d10ad8f520304ae1cad9b6c5a95372aef1b66b7bfb53a", size = 60029, upload-time = "2026-05-08T21:00:22.713Z" }, + { url = "https://files.pythonhosted.org/packages/70/60/189be62e0dd898dce3b331e1b8c7a543cd3a405ac0c81fe8ee8a9d5d77e1/propcache-0.5.2-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:45f11346f884bc47444f6e6647131055844134c3175b629f84952e2b5cd62b64", size = 56774, upload-time = "2026-05-08T21:00:24.001Z" }, + { url = "https://files.pythonhosted.org/packages/ea/9e/93377b9c7939c1ffae98f878dee955efadfd638078bc86dbc21f9d52f651/propcache-0.5.2-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:8e778ebd44ef4f66ed60a0416b06b489687db264a9c0b3620362f26489492913", size = 63532, upload-time = "2026-05-08T21:00:25.545Z" }, + { url = "https://files.pythonhosted.org/packages/14/f9/590ef6cfb9b8028d516d287812ece32bb0bc5f11fbb9c8bf6b2e6313fec8/propcache-0.5.2-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:c0cb9ed24c8964e172768d455a38254c2dd8a552905729ce006cad3d3dda59b1", size = 61592, upload-time = "2026-05-08T21:00:27.186Z" }, + { url = "https://files.pythonhosted.org/packages/b4/5e/70958b3034c297a630bba2f17ca7abc2d5f39a803ad7e370ab79d1ecd022/propcache-0.5.2-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:1d1ad32d9d4355e2be65574fd0bfd3677e7066b009cd5b9b2dee8aa6a6393b33", size = 64788, upload-time = "2026-05-08T21:00:28.8Z" }, + { url = "https://files.pythonhosted.org/packages/12/fd/77fe5936d8c3086ca9048f7f415f122ed82e53884a9ec193646b42deef06/propcache-0.5.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:c80f4ba3e8f00189165999a742ee526ebeccedf6c3f7beb0c7df821e9772435a", size = 62514, upload-time = "2026-05-08T21:00:30.098Z" }, + { url = "https://files.pythonhosted.org/packages/cf/74/66bd798b5b3be70aa1b391f5cc9d6a0a5532d7fd3b19ec0b213e72e6ad9d/propcache-0.5.2-cp312-cp312-win32.whl", hash = "sha256:8c7972d8f193740d9175f0998ab38717e6cd322d5935c5b0fef8c0d323fd9031", size = 39018, upload-time = "2026-05-08T21:00:31.622Z" }, + { url = "https://files.pythonhosted.org/packages/61/7c/5c0d34aa3024694d6dcb9271cdbdd08c4e47c1c0ad95ec7e7bc74cdea145/propcache-0.5.2-cp312-cp312-win_amd64.whl", hash = "sha256:d9ee8826a7d47863a08ac44e1a5f611a462eefc3a194b492da242128bec75b42", size = 42322, upload-time = "2026-05-08T21:00:32.918Z" }, + { url = "https://files.pythonhosted.org/packages/4d/91/875812f1a3feb20ceba818ef39fbe4d92f1081e04ac815c822496d0d038b/propcache-0.5.2-cp312-cp312-win_arm64.whl", hash = "sha256:2800a4a8ead6b28cccd1ec54b59346f0def7922ee1c7598e8499c733cfbb7c84", size = 38172, upload-time = "2026-05-08T21:00:35.124Z" }, + { url = "https://files.pythonhosted.org/packages/3a/ed/1cdcab6ba3d6ab7feca11fc14f0eeea80755bb53ef4e892079f31b10a25f/propcache-0.5.2-py3-none-any.whl", hash = "sha256:be1ddfcbb376e3de5d2e2db1d58d6d67463e6b4f9f040c000de8e300295465fe", size = 14036, upload-time = "2026-05-08T21:02:10.673Z" }, +] + +[[package]] +name = "protobuf" +version = "6.33.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/66/70/e908e9c5e52ef7c3a6c7902c9dfbb34c7e29c25d2f81ade3856445fd5c94/protobuf-6.33.6.tar.gz", hash = "sha256:a6768d25248312c297558af96a9f9c929e8c4cee0659cb07e780731095f38135", size = 444531, upload-time = "2026-03-18T19:05:00.988Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fc/9f/2f509339e89cfa6f6a4c4ff50438db9ca488dec341f7e454adad60150b00/protobuf-6.33.6-cp310-abi3-win32.whl", hash = "sha256:7d29d9b65f8afef196f8334e80d6bc1d5d4adedb449971fefd3723824e6e77d3", size = 425739, upload-time = "2026-03-18T19:04:48.373Z" }, + { url = "https://files.pythonhosted.org/packages/76/5d/683efcd4798e0030c1bab27374fd13a89f7c2515fb1f3123efdfaa5eab57/protobuf-6.33.6-cp310-abi3-win_amd64.whl", hash = "sha256:0cd27b587afca21b7cfa59a74dcbd48a50f0a6400cfb59391340ad729d91d326", size = 437089, upload-time = "2026-03-18T19:04:50.381Z" }, + { url = "https://files.pythonhosted.org/packages/5c/01/a3c3ed5cd186f39e7880f8303cc51385a198a81469d53d0fdecf1f64d929/protobuf-6.33.6-cp39-abi3-macosx_10_9_universal2.whl", hash = "sha256:9720e6961b251bde64edfdab7d500725a2af5280f3f4c87e57c0208376aa8c3a", size = 427737, upload-time = "2026-03-18T19:04:51.866Z" }, + { url = "https://files.pythonhosted.org/packages/ee/90/b3c01fdec7d2f627b3a6884243ba328c1217ed2d978def5c12dc50d328a3/protobuf-6.33.6-cp39-abi3-manylinux2014_aarch64.whl", hash = "sha256:e2afbae9b8e1825e3529f88d514754e094278bb95eadc0e199751cdd9a2e82a2", size = 324610, upload-time = "2026-03-18T19:04:53.096Z" }, + { url = "https://files.pythonhosted.org/packages/9b/ca/25afc144934014700c52e05103c2421997482d561f3101ff352e1292fb81/protobuf-6.33.6-cp39-abi3-manylinux2014_s390x.whl", hash = "sha256:c96c37eec15086b79762ed265d59ab204dabc53056e3443e702d2681f4b39ce3", size = 339381, upload-time = "2026-03-18T19:04:54.616Z" }, + { url = "https://files.pythonhosted.org/packages/16/92/d1e32e3e0d894fe00b15ce28ad4944ab692713f2e7f0a99787405e43533a/protobuf-6.33.6-cp39-abi3-manylinux2014_x86_64.whl", hash = "sha256:e9db7e292e0ab79dd108d7f1a94fe31601ce1ee3f7b79e0692043423020b0593", size = 323436, upload-time = "2026-03-18T19:04:55.768Z" }, + { url = "https://files.pythonhosted.org/packages/c4/72/02445137af02769918a93807b2b7890047c32bfb9f90371cbc12688819eb/protobuf-6.33.6-py3-none-any.whl", hash = "sha256:77179e006c476e69bf8e8ce866640091ec42e1beb80b213c3900006ecfba6901", size = 170656, upload-time = "2026-03-18T19:04:59.826Z" }, +] + +[[package]] +name = "pyarrow" +version = "25.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/27/f3/95428098d1fa7d04432fb750eed06b41304c2f6a5d3319985e64db2d9d41/pyarrow-25.0.0.tar.gz", hash = "sha256:d2d697008b5ec06d75952ef260c2e9a8a0f6ccfce24266c04c9c8ade927cb3b4", size = 1199181, upload-time = "2026-07-10T08:29:50.116Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/73/44/fdd3a4377807b7dcabe2d4b5aa99dbbc98e2e5df3f1ca4e7f0aec492d987/pyarrow-25.0.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:149730a3d1f0fb59d663a0b8aa210adfd9c17c27cd94a0d143e60daea8320d4e", size = 35850884, upload-time = "2026-07-10T08:26:47.357Z" }, + { url = "https://files.pythonhosted.org/packages/bf/71/9f053177a7709b8c90abb00a2375b916286f9f0d6cfb21a5cadd4ef811e8/pyarrow-25.0.0-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:0721332c30fdd453fdd1fc203b2ac1f4c9db5aea28fa38d41f2574c4b068b9ec", size = 37616197, upload-time = "2026-07-10T08:26:53.564Z" }, + { url = "https://files.pythonhosted.org/packages/95/1a/22bfb6597dcdc861fa83c39c06e1457cb56f698940eff42fbb25de30e8e5/pyarrow-25.0.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:fa1482b3da10cac2d4db6e26b81da543e237616af2ef6d466018b31ca586496f", size = 46841966, upload-time = "2026-07-10T08:27:07.685Z" }, + { url = "https://files.pythonhosted.org/packages/55/0e/cd705c042bc4fe7022478db577fcab4abdcfabb9bc37ab7a75556b3fcb2b/pyarrow-25.0.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:5d1dbf24e151042f2fa3c129563f65d66674128868496fb008c4272b16bdf778", size = 50088993, upload-time = "2026-07-10T08:27:14.268Z" }, + { url = "https://files.pythonhosted.org/packages/98/ee/d822e1ee31fe31ec5d057210e0605c950b975dcd8d9a332976cc859a9df8/pyarrow-25.0.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:20887a762dd61dcc530f93a140840ab1f6aa7836b33270e42d627ab3cf11e537", size = 49941005, upload-time = "2026-07-10T08:27:21.274Z" }, + { url = "https://files.pythonhosted.org/packages/33/1b/207a90cc64619a095eb75a263ae069735f2810056d43c667befd573ec083/pyarrow-25.0.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:58d1ab556b0cea1c93fdb799b24ad58adb2f2a2788dbce782a94f64ae1a5cc9b", size = 53112355, upload-time = "2026-07-10T08:27:27.911Z" }, + { url = "https://files.pythonhosted.org/packages/7e/fe/81d1e5f8beed15c01e98649d5c6e2167b67fd395884a2488f18bf1cf0dba/pyarrow-25.0.0-cp312-cp312-win_amd64.whl", hash = "sha256:3f356afe61186395c861d5cd63dc21ff7d5fa335012a4668d979257df7fea0f5", size = 27945954, upload-time = "2026-07-10T08:27:32.903Z" }, +] + +[[package]] +name = "pydantic" +version = "2.12.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "annotated-types" }, + { name = "pydantic-core" }, + { name = "typing-extensions" }, + { name = "typing-inspection" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/69/44/36f1a6e523abc58ae5f928898e4aca2e0ea509b5aa6f6f392a5d882be928/pydantic-2.12.5.tar.gz", hash = "sha256:4d351024c75c0f085a9febbb665ce8c0c6ec5d30e903bdb6394b7ede26aebb49", size = 821591, upload-time = "2025-11-26T15:11:46.471Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5a/87/b70ad306ebb6f9b585f114d0ac2137d792b48be34d732d60e597c2f8465a/pydantic-2.12.5-py3-none-any.whl", hash = "sha256:e561593fccf61e8a20fc46dfc2dfe075b8be7d0188df33f221ad1f0139180f9d", size = 463580, upload-time = "2025-11-26T15:11:44.605Z" }, +] + +[[package]] +name = "pydantic-core" +version = "2.41.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/71/70/23b021c950c2addd24ec408e9ab05d59b035b39d97cdc1130e1bce647bb6/pydantic_core-2.41.5.tar.gz", hash = "sha256:08daa51ea16ad373ffd5e7606252cc32f07bc72b28284b6bc9c6df804816476e", size = 460952, upload-time = "2025-11-04T13:43:49.098Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5f/5d/5f6c63eebb5afee93bcaae4ce9a898f3373ca23df3ccaef086d0233a35a7/pydantic_core-2.41.5-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:f41a7489d32336dbf2199c8c0a215390a751c5b014c2c1c5366e817202e9cdf7", size = 2110990, upload-time = "2025-11-04T13:39:58.079Z" }, + { url = "https://files.pythonhosted.org/packages/aa/32/9c2e8ccb57c01111e0fd091f236c7b371c1bccea0fa85247ac55b1e2b6b6/pydantic_core-2.41.5-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:070259a8818988b9a84a449a2a7337c7f430a22acc0859c6b110aa7212a6d9c0", size = 1896003, upload-time = "2025-11-04T13:39:59.956Z" }, + { url = "https://files.pythonhosted.org/packages/68/b8/a01b53cb0e59139fbc9e4fda3e9724ede8de279097179be4ff31f1abb65a/pydantic_core-2.41.5-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e96cea19e34778f8d59fe40775a7a574d95816eb150850a85a7a4c8f4b94ac69", size = 1919200, upload-time = "2025-11-04T13:40:02.241Z" }, + { url = "https://files.pythonhosted.org/packages/38/de/8c36b5198a29bdaade07b5985e80a233a5ac27137846f3bc2d3b40a47360/pydantic_core-2.41.5-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:ed2e99c456e3fadd05c991f8f437ef902e00eedf34320ba2b0842bd1c3ca3a75", size = 2052578, upload-time = "2025-11-04T13:40:04.401Z" }, + { url = "https://files.pythonhosted.org/packages/00/b5/0e8e4b5b081eac6cb3dbb7e60a65907549a1ce035a724368c330112adfdd/pydantic_core-2.41.5-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:65840751b72fbfd82c3c640cff9284545342a4f1eb1586ad0636955b261b0b05", size = 2208504, upload-time = "2025-11-04T13:40:06.072Z" }, + { url = "https://files.pythonhosted.org/packages/77/56/87a61aad59c7c5b9dc8caad5a41a5545cba3810c3e828708b3d7404f6cef/pydantic_core-2.41.5-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:e536c98a7626a98feb2d3eaf75944ef6f3dbee447e1f841eae16f2f0a72d8ddc", size = 2335816, upload-time = "2025-11-04T13:40:07.835Z" }, + { url = "https://files.pythonhosted.org/packages/0d/76/941cc9f73529988688a665a5c0ecff1112b3d95ab48f81db5f7606f522d3/pydantic_core-2.41.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:eceb81a8d74f9267ef4081e246ffd6d129da5d87e37a77c9bde550cb04870c1c", size = 2075366, upload-time = "2025-11-04T13:40:09.804Z" }, + { url = "https://files.pythonhosted.org/packages/d3/43/ebef01f69baa07a482844faaa0a591bad1ef129253ffd0cdaa9d8a7f72d3/pydantic_core-2.41.5-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:d38548150c39b74aeeb0ce8ee1d8e82696f4a4e16ddc6de7b1d8823f7de4b9b5", size = 2171698, upload-time = "2025-11-04T13:40:12.004Z" }, + { url = "https://files.pythonhosted.org/packages/b1/87/41f3202e4193e3bacfc2c065fab7706ebe81af46a83d3e27605029c1f5a6/pydantic_core-2.41.5-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:c23e27686783f60290e36827f9c626e63154b82b116d7fe9adba1fda36da706c", size = 2132603, upload-time = "2025-11-04T13:40:13.868Z" }, + { url = "https://files.pythonhosted.org/packages/49/7d/4c00df99cb12070b6bccdef4a195255e6020a550d572768d92cc54dba91a/pydantic_core-2.41.5-cp312-cp312-musllinux_1_1_armv7l.whl", hash = "sha256:482c982f814460eabe1d3bb0adfdc583387bd4691ef00b90575ca0d2b6fe2294", size = 2329591, upload-time = "2025-11-04T13:40:15.672Z" }, + { url = "https://files.pythonhosted.org/packages/cc/6a/ebf4b1d65d458f3cda6a7335d141305dfa19bdc61140a884d165a8a1bbc7/pydantic_core-2.41.5-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:bfea2a5f0b4d8d43adf9d7b8bf019fb46fdd10a2e5cde477fbcb9d1fa08c68e1", size = 2319068, upload-time = "2025-11-04T13:40:17.532Z" }, + { url = "https://files.pythonhosted.org/packages/49/3b/774f2b5cd4192d5ab75870ce4381fd89cf218af999515baf07e7206753f0/pydantic_core-2.41.5-cp312-cp312-win32.whl", hash = "sha256:b74557b16e390ec12dca509bce9264c3bbd128f8a2c376eaa68003d7f327276d", size = 1985908, upload-time = "2025-11-04T13:40:19.309Z" }, + { url = "https://files.pythonhosted.org/packages/86/45/00173a033c801cacf67c190fef088789394feaf88a98a7035b0e40d53dc9/pydantic_core-2.41.5-cp312-cp312-win_amd64.whl", hash = "sha256:1962293292865bca8e54702b08a4f26da73adc83dd1fcf26fbc875b35d81c815", size = 2020145, upload-time = "2025-11-04T13:40:21.548Z" }, + { url = "https://files.pythonhosted.org/packages/f9/22/91fbc821fa6d261b376a3f73809f907cec5ca6025642c463d3488aad22fb/pydantic_core-2.41.5-cp312-cp312-win_arm64.whl", hash = "sha256:1746d4a3d9a794cacae06a5eaaccb4b8643a131d45fbc9af23e353dc0a5ba5c3", size = 1976179, upload-time = "2025-11-04T13:40:23.393Z" }, + { url = "https://files.pythonhosted.org/packages/09/32/59b0c7e63e277fa7911c2fc70ccfb45ce4b98991e7ef37110663437005af/pydantic_core-2.41.5-graalpy312-graalpy250_312_native-macosx_10_12_x86_64.whl", hash = "sha256:7da7087d756b19037bc2c06edc6c170eeef3c3bafcb8f532ff17d64dc427adfd", size = 2110495, upload-time = "2025-11-04T13:42:49.689Z" }, + { url = "https://files.pythonhosted.org/packages/aa/81/05e400037eaf55ad400bcd318c05bb345b57e708887f07ddb2d20e3f0e98/pydantic_core-2.41.5-graalpy312-graalpy250_312_native-macosx_11_0_arm64.whl", hash = "sha256:aabf5777b5c8ca26f7824cb4a120a740c9588ed58df9b2d196ce92fba42ff8dc", size = 1915388, upload-time = "2025-11-04T13:42:52.215Z" }, + { url = "https://files.pythonhosted.org/packages/6e/0d/e3549b2399f71d56476b77dbf3cf8937cec5cd70536bdc0e374a421d0599/pydantic_core-2.41.5-graalpy312-graalpy250_312_native-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c007fe8a43d43b3969e8469004e9845944f1a80e6acd47c150856bb87f230c56", size = 1942879, upload-time = "2025-11-04T13:42:56.483Z" }, + { url = "https://files.pythonhosted.org/packages/f7/07/34573da085946b6a313d7c42f82f16e8920bfd730665de2d11c0c37a74b5/pydantic_core-2.41.5-graalpy312-graalpy250_312_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:76d0819de158cd855d1cbb8fcafdf6f5cf1eb8e470abe056d5d161106e38062b", size = 2139017, upload-time = "2025-11-04T13:42:59.471Z" }, +] + +[[package]] +name = "pygments" +version = "2.20.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c3/b2/bc9c9196916376152d655522fdcebac55e66de6603a76a02bca1b6414f6c/pygments-2.20.0.tar.gz", hash = "sha256:6757cd03768053ff99f3039c1a36d6c0aa0b263438fcab17520b30a303a82b5f", size = 4955991, upload-time = "2026-03-29T13:29:33.898Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f4/7e/a72dd26f3b0f4f2bf1dd8923c85f7ceb43172af56d63c7383eb62b332364/pygments-2.20.0-py3-none-any.whl", hash = "sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176", size = 1231151, upload-time = "2026-03-29T13:29:30.038Z" }, +] + +[[package]] +name = "python-dateutil" +version = "2.9.0.post0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "six" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/66/c0/0c8b6ad9f17a802ee498c46e004a0eb49bc148f2fd230864601a86dcf6db/python-dateutil-2.9.0.post0.tar.gz", hash = "sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3", size = 342432, upload-time = "2024-03-01T18:36:20.211Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl", hash = "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427", size = 229892, upload-time = "2024-03-01T18:36:18.57Z" }, +] + +[[package]] +name = "python-discovery" +version = "1.4.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "filelock" }, + { name = "platformdirs" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/4c/81/58c70036dffeccb7fe7d79d6260c69f7a28272bbd3909c29a01ea9422744/python_discovery-1.4.4.tar.gz", hash = "sha256:5cad33982d412c1f3ffb8f9ca4ea292c9680bca3942451d30b69c37fce53a4a3", size = 72212, upload-time = "2026-07-08T23:06:50.691Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9d/ae/84bc0d2440c95772272bb6f4b3d09ccf08b2898fce89b3d4f969a9fc74e9/python_discovery-1.4.4-py3-none-any.whl", hash = "sha256:abebe9120b43453b68c908acfb1e72a19d1a959ed2cb620ad38fc57d08056dbe", size = 34181, upload-time = "2026-07-08T23:06:49.402Z" }, +] + +[[package]] +name = "python-dotenv" +version = "1.2.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/82/ed/0301aeeac3e5353ef3d94b6ec08bbcabd04a72018415dcb29e588514bba8/python_dotenv-1.2.2.tar.gz", hash = "sha256:2c371a91fbd7ba082c2c1dc1f8bf89ca22564a087c2c287cd9b662adde799cf3", size = 50135, upload-time = "2026-03-01T16:00:26.196Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0b/d7/1959b9648791274998a9c3526f6d0ec8fd2233e4d4acce81bbae76b44b2a/python_dotenv-1.2.2-py3-none-any.whl", hash = "sha256:1d8214789a24de455a8b8bd8ae6fe3c6b69a5e3d64aa8a8e5d68e694bbcb285a", size = 22101, upload-time = "2026-03-01T16:00:25.09Z" }, +] + +[[package]] +name = "pywin32" +version = "312" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/83/ff/32aa7d2ed0ab12b323aaa64f9b75e6ad4f8fd09f9ccfc28c79414d46838d/pywin32-312-cp312-cp312-win32.whl", hash = "sha256:dab4f65ac9c4e48400a2a0530c46c3c579cd5905ecd11b80692373915269208b", size = 6371877, upload-time = "2026-06-04T07:49:28.836Z" }, + { url = "https://files.pythonhosted.org/packages/03/d9/77040d3b43df3f3be32ea289433d660d2727f5ba327bc73be835127d9d60/pywin32-312-cp312-cp312-win_amd64.whl", hash = "sha256:b457f6d628a47e8a7346ce22acb7e1a46a4a78b52e1d17e1af56871bd19a93bc", size = 6914841, upload-time = "2026-06-04T07:49:31.85Z" }, + { url = "https://files.pythonhosted.org/packages/e3/cc/7b1ec671775756020a0ee7f4feeaf3c568f0ab86bd3900088cf986937a92/pywin32-312-cp312-cp312-win_arm64.whl", hash = "sha256:6017c58e12f6809fbb0555b75df144c2922a9ffd18e4b9b5afa863b6c1a9d950", size = 6727901, upload-time = "2026-06-04T07:49:34.244Z" }, +] + +[[package]] +name = "pyyaml" +version = "6.0.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/05/8e/961c0007c59b8dd7729d542c61a4d537767a59645b82a0b521206e1e25c2/pyyaml-6.0.3.tar.gz", hash = "sha256:d76623373421df22fb4cf8817020cbb7ef15c725b9d5e45f17e189bfc384190f", size = 130960, upload-time = "2025-09-25T21:33:16.546Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/33/422b98d2195232ca1826284a76852ad5a86fe23e31b009c9886b2d0fb8b2/pyyaml-6.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:7f047e29dcae44602496db43be01ad42fc6f1cc0d8cd6c83d342306c32270196", size = 182063, upload-time = "2025-09-25T21:32:11.445Z" }, + { url = "https://files.pythonhosted.org/packages/89/a0/6cf41a19a1f2f3feab0e9c0b74134aa2ce6849093d5517a0c550fe37a648/pyyaml-6.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:fc09d0aa354569bc501d4e787133afc08552722d3ab34836a80547331bb5d4a0", size = 173973, upload-time = "2025-09-25T21:32:12.492Z" }, + { url = "https://files.pythonhosted.org/packages/ed/23/7a778b6bd0b9a8039df8b1b1d80e2e2ad78aa04171592c8a5c43a56a6af4/pyyaml-6.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9149cad251584d5fb4981be1ecde53a1ca46c891a79788c0df828d2f166bda28", size = 775116, upload-time = "2025-09-25T21:32:13.652Z" }, + { url = "https://files.pythonhosted.org/packages/65/30/d7353c338e12baef4ecc1b09e877c1970bd3382789c159b4f89d6a70dc09/pyyaml-6.0.3-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5fdec68f91a0c6739b380c83b951e2c72ac0197ace422360e6d5a959d8d97b2c", size = 844011, upload-time = "2025-09-25T21:32:15.21Z" }, + { url = "https://files.pythonhosted.org/packages/8b/9d/b3589d3877982d4f2329302ef98a8026e7f4443c765c46cfecc8858c6b4b/pyyaml-6.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ba1cc08a7ccde2d2ec775841541641e4548226580ab850948cbfda66a1befcdc", size = 807870, upload-time = "2025-09-25T21:32:16.431Z" }, + { url = "https://files.pythonhosted.org/packages/05/c0/b3be26a015601b822b97d9149ff8cb5ead58c66f981e04fedf4e762f4bd4/pyyaml-6.0.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:8dc52c23056b9ddd46818a57b78404882310fb473d63f17b07d5c40421e47f8e", size = 761089, upload-time = "2025-09-25T21:32:17.56Z" }, + { url = "https://files.pythonhosted.org/packages/be/8e/98435a21d1d4b46590d5459a22d88128103f8da4c2d4cb8f14f2a96504e1/pyyaml-6.0.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:41715c910c881bc081f1e8872880d3c650acf13dfa8214bad49ed4cede7c34ea", size = 790181, upload-time = "2025-09-25T21:32:18.834Z" }, + { url = "https://files.pythonhosted.org/packages/74/93/7baea19427dcfbe1e5a372d81473250b379f04b1bd3c4c5ff825e2327202/pyyaml-6.0.3-cp312-cp312-win32.whl", hash = "sha256:96b533f0e99f6579b3d4d4995707cf36df9100d67e0c8303a0c55b27b5f99bc5", size = 137658, upload-time = "2025-09-25T21:32:20.209Z" }, + { url = "https://files.pythonhosted.org/packages/86/bf/899e81e4cce32febab4fb42bb97dcdf66bc135272882d1987881a4b519e9/pyyaml-6.0.3-cp312-cp312-win_amd64.whl", hash = "sha256:5fcd34e47f6e0b794d17de1b4ff496c00986e1c83f7ab2fb8fcfe9616ff7477b", size = 154003, upload-time = "2025-09-25T21:32:21.167Z" }, + { url = "https://files.pythonhosted.org/packages/1a/08/67bd04656199bbb51dbed1439b7f27601dfb576fb864099c7ef0c3e55531/pyyaml-6.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:64386e5e707d03a7e172c0701abfb7e10f0fb753ee1d773128192742712a98fd", size = 140344, upload-time = "2025-09-25T21:32:22.617Z" }, +] + +[[package]] +name = "referencing" +version = "0.37.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "attrs" }, + { name = "rpds-py" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/22/f5/df4e9027acead3ecc63e50fe1e36aca1523e1719559c499951bb4b53188f/referencing-0.37.0.tar.gz", hash = "sha256:44aefc3142c5b842538163acb373e24cce6632bd54bdb01b21ad5863489f50d8", size = 78036, upload-time = "2025-10-13T15:30:48.871Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2c/58/ca301544e1fa93ed4f80d724bf5b194f6e4b945841c5bfd555878eea9fcb/referencing-0.37.0-py3-none-any.whl", hash = "sha256:381329a9f99628c9069361716891d34ad94af76e461dcb0335825aecc7692231", size = 26766, upload-time = "2025-10-13T15:30:47.625Z" }, +] + +[[package]] +name = "regex" +version = "2026.7.10" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7b/37/451aaddbf50922f34d744ad5ca919ae1fcfac112123885d9728f52a484b3/regex-2026.7.10.tar.gz", hash = "sha256:1050fedf0a8a92e843971120c2f57c3a99bea86c0dfa1d63a9fac053fe54b135", size = 416282, upload-time = "2026-07-10T19:49:46.267Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b3/9c/2503d4ccf3452dc323f8baa3cf3ee10406037d52735c76cfced81423f183/regex-2026.7.10-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:7252b48b0c60100095088fbeb281fca9a4fcf678a4e04b1c520c3f8613c952c4", size = 497114, upload-time = "2026-07-10T19:47:16.22Z" }, + { url = "https://files.pythonhosted.org/packages/91/eb/04534f4263a4f658cd20a511e9d6124350044f2214eb24fee2db96acf318/regex-2026.7.10-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:da6ef4cb8d457aab0482b50120136ae94238aaa421863eaa7d599759742c72d6", size = 297422, upload-time = "2026-07-10T19:47:17.794Z" }, + { url = "https://files.pythonhosted.org/packages/ca/2d/35809de392ab66ba439b58c3187ae3b8b53c883233f284b59961e5725c99/regex-2026.7.10-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:fe7ff456c22725c9d9017f7a2a7df2b51af6df77314176760b22e2d05278e181", size = 292110, upload-time = "2026-07-10T19:47:19.188Z" }, + { url = "https://files.pythonhosted.org/packages/ad/1e/5ce0fbe9aab071893ce2b7df020d0f561f7b411ec334124302468d587884/regex-2026.7.10-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f3463a5f26be513a49e4d497debcf1b252a2db7b92c77d89621aa90b83d2dd38", size = 796800, upload-time = "2026-07-10T19:47:20.639Z" }, + { url = "https://files.pythonhosted.org/packages/d4/67/c1ccbada395c10e334763b583e1039b1660b142303ebb941d4269130b22f/regex-2026.7.10-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:948dfc62683a6947b9b486c4598d8f6e3ecc542478b6767b87d52be68aeb55c6", size = 865509, upload-time = "2026-07-10T19:47:22.135Z" }, + { url = "https://files.pythonhosted.org/packages/0e/06/f0b31afc16c1208f945b66290eb2a9936ab8becdfb23bbcedb91cc5f9d9b/regex-2026.7.10-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:c2cbd385d82f63bb35edb60b09b08abad3619bd0a4a492ae59e55afaf98e1b9d", size = 912395, upload-time = "2026-07-10T19:47:24.128Z" }, + { url = "https://files.pythonhosted.org/packages/0b/1c/8687de3a6c3220f4f872a9bf4bcd8dc249f2a96e7dddfa93de8bd4d16399/regex-2026.7.10-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f6222cafe00e072bb2b8f14142cd969637411fbc4dd3b1d73a90a3b817fa046f", size = 801308, upload-time = "2026-07-10T19:47:25.696Z" }, + { url = "https://files.pythonhosted.org/packages/5f/e3/60a40ec02a2315d826414a125640aceb6f30450574c530c8f352110ece0e/regex-2026.7.10-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:65ee5d1ac3cd541325f5ac92625b1c1505f4d171520dd931bda7952895c5321a", size = 777120, upload-time = "2026-07-10T19:47:27.158Z" }, + { url = "https://files.pythonhosted.org/packages/6a/9a/ec579b4f840ac59bc7c192b56e66abd4cbf385615300d59f7c94bf6863ae/regex-2026.7.10-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:aa34473fbcc108fea403074f3f45091461b18b2047d136f16ffaa4c65ad46a68", size = 785164, upload-time = "2026-07-10T19:47:28.732Z" }, + { url = "https://files.pythonhosted.org/packages/ad/1c/60d88afd5f98d4b0fb1f8b8969270628140dc01c7ff93a939f2aa83f31a6/regex-2026.7.10-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:9d028d189d8f38d7ff292f22187c0df37f2317f554d2ed9a2908ada330af57c0", size = 860161, upload-time = "2026-07-10T19:47:30.605Z" }, + { url = "https://files.pythonhosted.org/packages/2a/40/08ae3ba45fe79e48c9a888a3389a7ee7e2d8c580d2d996da5ece02dfdcb9/regex-2026.7.10-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:396ea70e4ea1f19571940add3bad9fd3eb6a19dc610d0d01f692bc1ba0c10cb4", size = 765829, upload-time = "2026-07-10T19:47:32.06Z" }, + { url = "https://files.pythonhosted.org/packages/12/e6/e613c6755d19aca9d977cdc3418a1991ffc8f386779752dd8fdfa888ea89/regex-2026.7.10-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:ebbf0d83ed5271991d666e54bb6c90ac2c55fb2ef3a88740c6af85dc85de2402", size = 852170, upload-time = "2026-07-10T19:47:33.567Z" }, + { url = "https://files.pythonhosted.org/packages/03/33/89072f2060e6b844b4916d5bc40ef01e973640c703025707869264ec75ab/regex-2026.7.10-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:58a4571b2a093f6f6ee4fd281faa8ebf645abcf575f758173ea2605c7a1e1ecb", size = 789550, upload-time = "2026-07-10T19:47:35.395Z" }, + { url = "https://files.pythonhosted.org/packages/e3/3c/4bc8be9a155035e63780ccac1da101f36194946fdc3f6fce90c7179fc6df/regex-2026.7.10-cp312-cp312-win32.whl", hash = "sha256:eac1207936555aa691ce32df1432b478f2729d54e6d93a1f4db9215bcd8eb47d", size = 267151, upload-time = "2026-07-10T19:47:37.047Z" }, + { url = "https://files.pythonhosted.org/packages/35/73/9f5aade65bb98cc6e99c336e45a49a658300720c16721f3e687f8d754fec/regex-2026.7.10-cp312-cp312-win_amd64.whl", hash = "sha256:ecae626449d00db8c08f8f1fc00047a32d6d7eb5402b3976f5c3fda2b80a7a4f", size = 277751, upload-time = "2026-07-10T19:47:38.488Z" }, + { url = "https://files.pythonhosted.org/packages/36/6f/d069dd12872ea1d50e17319d342f89e2072cae4b62f4245009a1108c74d8/regex-2026.7.10-cp312-cp312-win_arm64.whl", hash = "sha256:87794549a3f5c1c2bdfba2380c1bf87b931e375f4133d929da44f95e396bf5fe", size = 277063, upload-time = "2026-07-10T19:47:40.023Z" }, +] + +[[package]] +name = "requests" +version = "2.34.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "certifi" }, + { name = "charset-normalizer" }, + { name = "idna" }, + { name = "urllib3" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/ac/c3/e2a2b89f2d3e2179abd6d00ebd70bff6273f37fb3e0cc209f48b39d00cbf/requests-2.34.2.tar.gz", hash = "sha256:f288924cae4e29463698d6d60bc6a4da69c89185ad1e0bcc4104f584e960b9ed", size = 142856, upload-time = "2026-05-14T19:25:27.735Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a0/f4/c67b0b3f1b9245e8d266f0f112c500d50e5b4e83cb6f3b71b6528104182a/requests-2.34.2-py3-none-any.whl", hash = "sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0", size = 73075, upload-time = "2026-05-14T19:25:26.443Z" }, +] + +[[package]] +name = "rich" +version = "15.0.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markdown-it-py" }, + { name = "pygments" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/c0/8f/0722ca900cc807c13a6a0c696dacf35430f72e0ec571c4275d2371fca3e9/rich-15.0.0.tar.gz", hash = "sha256:edd07a4824c6b40189fb7ac9bc4c52536e9780fbbfbddf6f1e2502c31b068c36", size = 230680, upload-time = "2026-04-12T08:24:00.75Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/82/3b/64d4899d73f91ba49a8c18a8ff3f0ea8f1c1d75481760df8c68ef5235bf5/rich-15.0.0-py3-none-any.whl", hash = "sha256:33bd4ef74232fb73fe9279a257718407f169c09b78a87ad3d296f548e27de0bb", size = 310654, upload-time = "2026-04-12T08:24:02.83Z" }, +] + +[[package]] +name = "rpds-py" +version = "2026.6.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/aa/2a/9618a122aeb2a169a28b03889a2995fe297588964333d4a7d67bdf46e147/rpds_py-2026.6.3.tar.gz", hash = "sha256:1cebd1337c242e4ec2293e541f712b2da849b29f48f0c293684b71c0632625d4", size = 64051, upload-time = "2026-06-30T07:17:53.009Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5c/be/2e8974163072e7bab7df1a5acd54c4498e75e35d6d18b864d3a9d5dadc92/rpds_py-2026.6.3-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:a0811d33247c3d6128a3001d763f2aa056bb3425204335400ac54f89eec3a0d0", size = 343691, upload-time = "2026-06-30T07:15:14.96Z" }, + { url = "https://files.pythonhosted.org/packages/a4/73/319dfa745dd668efe89309141ded489126461fcecd2b8f3a3cda185129b6/rpds_py-2026.6.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:538949e262e46caa31ac01bdb3c1e8f642622922cacbabbae6a8445d9dc33eaf", size = 338542, upload-time = "2026-06-30T07:15:16.267Z" }, + { url = "https://files.pythonhosted.org/packages/21/63/4239893be1c4d09b709b1a8f6be4188f0870084ff547f46606b8a75f1b03/rpds_py-2026.6.3-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:55927d532399c2c646100ff7feb48eaa940ad70f42cd68e1328f3ded9f81ca24", size = 368180, upload-time = "2026-06-30T07:15:17.62Z" }, + { url = "https://files.pythonhosted.org/packages/1c/ca/9c5de382225234ceb37b1844ebdb140db12b2a278bb9efe2fcd19f6c82ce/rpds_py-2026.6.3-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:f56f1695bc5c0871cbc33dc0130fcf503aab0c57dcc5a6700a4f49eba4f2652e", size = 375067, upload-time = "2026-06-30T07:15:18.952Z" }, + { url = "https://files.pythonhosted.org/packages/87/dc/863f69d1bf04ade34b7fe0d59b9fdf6f0135fe2d7cbca74f1d665589559d/rpds_py-2026.6.3-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:270b293dae9058fc9fcedab50f13cebf46fb8ed1d1d54e0521a9da5d6b211975", size = 490509, upload-time = "2026-06-30T07:15:20.434Z" }, + { url = "https://files.pythonhosted.org/packages/ce/ef/eac16a12048b45ec7c7fa94f2be3438a5f26bf9cc8580b18a1cfd609b7f6/rpds_py-2026.6.3-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:127565fead0a10943b282957bd5447804ff3160ad79f2ad2635e6d249e380680", size = 382754, upload-time = "2026-06-30T07:15:21.831Z" }, + { url = "https://files.pythonhosted.org/packages/04/8f/d2f3f532616be4d06c316ef119683e832bd3d41e112bf3a88f4151c95b17/rpds_py-2026.6.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ecabd69db66de867690f9797f2f8fa27ba501bbc24540cbdbdc649cd15888ba6", size = 366189, upload-time = "2026-06-30T07:15:23.371Z" }, + { url = "https://files.pythonhosted.org/packages/e3/29/41a7b0e98a4b44cd676ab7598419623373eb43b20be68c084935c1a8cf88/rpds_py-2026.6.3-cp312-cp312-manylinux_2_31_riscv64.whl", hash = "sha256:58eadac9cd119677b60e1cf8ac4052f35949d71b8a9e5556efccbe82533cf22a", size = 377750, upload-time = "2026-06-30T07:15:24.659Z" }, + { url = "https://files.pythonhosted.org/packages/2e/05/ecda0bec46f9a1565090bcdc941d023f6a25aff85fda28f89f8d19878152/rpds_py-2026.6.3-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:7491ee23305ac3eb59e492b6945881f5cd77a6f731061a3f25b77fd40f9e99a4", size = 395576, upload-time = "2026-06-30T07:15:25.987Z" }, + { url = "https://files.pythonhosted.org/packages/68/a8/6ed52f03ee6cb854ce78785cc9a9a672eb880e83fd7224d471f667d151f1/rpds_py-2026.6.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:2c99f7e8ccb3dd6e3e4bfeac657a7b208c9bac8075f4b078c02d7404c34107fa", size = 543807, upload-time = "2026-06-30T07:15:27.356Z" }, + { url = "https://files.pythonhosted.org/packages/8f/d6/156c0d3eea27ba09b92562ba2364ba124c0a061b199e17eac637cd25a5e2/rpds_py-2026.6.3-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:62698275682bf121181861295c9181e789030a2d516071f5b8f3c23c170cd0fc", size = 611187, upload-time = "2026-06-30T07:15:28.931Z" }, + { url = "https://files.pythonhosted.org/packages/f1/31/774212ed989c62f7f310220089f9b0a3fb8f40f5443d1727abd5d9f52bc9/rpds_py-2026.6.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:a214c993455f99a89aaeadc9b21241900037adc9d97203e374d75513c5911822", size = 573030, upload-time = "2026-06-30T07:15:30.553Z" }, + { url = "https://files.pythonhosted.org/packages/c9/50/22f73127a41f1ce4f87fe39aadfb9a126345801c274aa93ae88456249327/rpds_py-2026.6.3-cp312-cp312-win32.whl", hash = "sha256:501f9f04a588d6a09179368c57071301445191767c64e4b52a6aa9871f1ef5ed", size = 202185, upload-time = "2026-06-30T07:15:32.027Z" }, + { url = "https://files.pythonhosted.org/packages/04/3a/f0ee4d4dde9d3b69dedf1b5f74e7a40017046d55052d173e418c6a94f960/rpds_py-2026.6.3-cp312-cp312-win_amd64.whl", hash = "sha256:2c958bf94822e9290a40aaf2a822d4bc5c88099093e3948ad6c571eca9272e5f", size = 220394, upload-time = "2026-06-30T07:15:33.359Z" }, + { url = "https://files.pythonhosted.org/packages/f3/83/3382fe37f809b59f02aac04dbc4e765b480b46ee0227ed516e3bdc4d3dfc/rpds_py-2026.6.3-cp312-cp312-win_arm64.whl", hash = "sha256:22bffe6042b9bcb0822bcd1955ec00e245daf17b4344e4ed8e9551b976b63e96", size = 215753, upload-time = "2026-06-30T07:15:34.778Z" }, +] + +[[package]] +name = "shellingham" +version = "1.5.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/58/15/8b3609fd3830ef7b27b655beb4b4e9c62313a4e8da8c676e142cc210d58e/shellingham-1.5.4.tar.gz", hash = "sha256:8dbca0739d487e5bd35ab3ca4b36e11c4078f3a234bfce294b0a0291363404de", size = 10310, upload-time = "2023-10-24T04:13:40.426Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl", hash = "sha256:7ecfff8f2fd72616f7481040475a65b2bf8af90a56c89140852d1120324e8686", size = 9755, upload-time = "2023-10-24T04:13:38.866Z" }, +] + +[[package]] +name = "six" +version = "1.17.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/94/e7/b2c673351809dca68a0e064b6af791aa332cf192da575fd474ed7d6f16a2/six-1.17.0.tar.gz", hash = "sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81", size = 34031, upload-time = "2024-12-04T17:35:28.174Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050, upload-time = "2024-12-04T17:35:26.475Z" }, +] + +[[package]] +name = "smmap" +version = "5.0.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/1f/ea/49c993d6dfdd7338c9b1000a0f36817ed7ec84577ae2e52f890d1a4ff909/smmap-5.0.3.tar.gz", hash = "sha256:4d9debb8b99007ae47165abc08670bd74cb74b5227dda7f643eccc4e9eb5642c", size = 22506, upload-time = "2026-03-09T03:43:26.1Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c1/d4/59e74daffcb57a07668852eeeb6035af9f32cbfd7a1d2511f17d2fe6a738/smmap-5.0.3-py3-none-any.whl", hash = "sha256:c106e05d5a61449cf6ba9a1e650227ecfb141590d2a98412103ff35d89fc7b2f", size = 24390, upload-time = "2026-03-09T03:43:24.361Z" }, +] + +[[package]] +name = "sniffio" +version = "1.3.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a2/87/a6771e1546d97e7e041b6ae58d80074f81b7d5121207425c964ddf5cfdbd/sniffio-1.3.1.tar.gz", hash = "sha256:f4324edc670a0f49750a81b895f35c3adb843cca46f0530f79fc1babb23789dc", size = 20372, upload-time = "2024-02-25T23:20:04.057Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e9/44/75a9c9421471a6c4805dbf2356f7c181a29c1879239abab1ea2cc8f38b40/sniffio-1.3.1-py3-none-any.whl", hash = "sha256:2f6da418d1f1e0fddd844478f41680e794e6051915791a034ff65e5f100525a2", size = 10235, upload-time = "2024-02-25T23:20:01.196Z" }, +] + +[[package]] +name = "soupsieve" +version = "2.8.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/47/2c/0a5f6f8ee0d5589e48c7640213ed5175d52cf540a06725b628cc1a45d6ce/soupsieve-2.8.4.tar.gz", hash = "sha256:e121fd02e975c695e4e9e8774a5ee35d74714b59307868dcc5319ad2d9e3328e", size = 121110, upload-time = "2026-05-24T13:55:57.154Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5e/f5/0c41cb68dcae6b7de4fac4188a3a9589e21fb31df21ea3a2e888db95e6c9/soupsieve-2.8.4-py3-none-any.whl", hash = "sha256:e7e6b0769c8f51ed59acab6e994b00621096cfb1c640a7509295987388fbaf65", size = 37304, upload-time = "2026-05-24T13:55:55.406Z" }, +] + +[[package]] +name = "swebench" +version = "4.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "beautifulsoup4" }, + { name = "chardet" }, + { name = "datasets" }, + { name = "docker" }, + { name = "ghapi" }, + { name = "gitpython" }, + { name = "modal" }, + { name = "pre-commit" }, + { name = "python-dotenv" }, + { name = "requests" }, + { name = "rich" }, + { name = "tenacity" }, + { name = "tqdm" }, + { name = "unidiff" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/24/e1/c997299ad7bf088876d30398203aa1eed7dec897670dc1aa35b1d748ffcc/swebench-4.1.0.tar.gz", hash = "sha256:5aaa6a92c2db1aa64892d28a47483ca46a45a15cf1d2df673d7744f71811dc9a", size = 134341, upload-time = "2025-09-11T02:58:00.447Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/36/67/981d8b642ac3eac7c8a7b7832ff8b2fb74f96b28b5fcd9a8979879e5c46d/swebench-4.1.0-py3-none-any.whl", hash = "sha256:1243776f720047cc9e20a427f7a52b75c13a07abda6154fb60fe77f82ec8af57", size = 157231, upload-time = "2025-09-11T02:57:58.953Z" }, +] + +[[package]] +name = "swebench-service" +version = "0.1.0" +source = { editable = "." } +dependencies = [ + { name = "aiohttp" }, + { name = "litellm" }, + { name = "mini-swe-agent" }, + { name = "msgspec" }, + { name = "pydantic" }, + { name = "pyyaml" }, + { name = "swebench" }, +] + +[package.metadata] +requires-dist = [ + { name = "aiohttp", specifier = "==3.14.1" }, + { name = "litellm", specifier = "==1.91.0" }, + { name = "mini-swe-agent", specifier = "==2.3.0" }, + { name = "msgspec", specifier = "==0.20.0" }, + { name = "pydantic", specifier = "==2.12.5" }, + { name = "pyyaml", specifier = "==6.0.3" }, + { name = "swebench", specifier = "==4.1.0" }, +] + +[[package]] +name = "synchronicity" +version = "0.12.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/5d/1c/f51dc54bbd302991026a53f9790735540e0e9e1184e9d5939f02446aa5bc/synchronicity-0.12.5.tar.gz", hash = "sha256:94d96b1d85698e3056b96a793b8c0949af6584e4a7d877fabdeb5385efe230aa", size = 60745, upload-time = "2026-06-18T21:06:23.545Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/06/74/ad9b99520f70c0bc3318e582e359d360cfc0f7afd7bf368a7f24013cece7/synchronicity-0.12.5-py3-none-any.whl", hash = "sha256:fdbbb10d437bc08a6b0f814fc66fddd1b58ffed314533d42f1ab555801e781af", size = 41107, upload-time = "2026-06-18T21:06:22.505Z" }, +] + +[[package]] +name = "tenacity" +version = "9.1.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/47/c6/ee486fd809e357697ee8a44d3d69222b344920433d3b6666ccd9b374630c/tenacity-9.1.4.tar.gz", hash = "sha256:adb31d4c263f2bd041081ab33b498309a57c77f9acf2db65aadf0898179cf93a", size = 49413, upload-time = "2026-02-07T10:45:33.841Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d7/c1/eb8f9debc45d3b7918a32ab756658a0904732f75e555402972246b0b8e71/tenacity-9.1.4-py3-none-any.whl", hash = "sha256:6095a360c919085f28c6527de529e76a06ad89b23659fa881ae0649b867a9d55", size = 28926, upload-time = "2026-02-07T10:45:32.24Z" }, +] + +[[package]] +name = "textual" +version = "8.2.8" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markdown-it-py", extra = ["linkify"] }, + { name = "mdit-py-plugins" }, + { name = "platformdirs" }, + { name = "pygments" }, + { name = "rich" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/00/21/39a76b01bd5eea82a04baaca7580e105d8c59450df03998345bb2cfb307b/textual-8.2.8.tar.gz", hash = "sha256:3f106a9fbc73e39dd266c9712432087de78a6d644084c7c241d6a25c3169115b", size = 1860502, upload-time = "2026-06-30T06:51:24.495Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fb/be/35261223d9416a0751cdff1c7b4a6f881387218a12d439fe22fefebc8c04/textual-8.2.8-py3-none-any.whl", hash = "sha256:267375fd402dc8d981457212efa71f0e3365fd17bba144ba9bb3ed7563cb374a", size = 731418, upload-time = "2026-06-30T06:51:26.364Z" }, +] + +[[package]] +name = "tiktoken" +version = "0.13.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "regex" }, + { name = "requests" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/e4/e5/5f3cb2159769d0f4324c0e9e87f9de3c4b1cd45848a96b2eb3566ad5ca77/tiktoken-0.13.0.tar.gz", hash = "sha256:c9435714c3a84c2319499de9a300c0e604449dd0799ff246458b3bb6a7f433c1", size = 38986, upload-time = "2026-05-15T04:51:27.153Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/85/8e/144bde4e01df66b34bb865557c7cd754ed08b036217ebd79c9db5e9048a9/tiktoken-0.13.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:32ac870a806cfb260a02d0cb70426aef02e038297f8ad50df5040bb5af360791", size = 1034888, upload-time = "2026-05-15T04:50:31.579Z" }, + { url = "https://files.pythonhosted.org/packages/36/18/d4ac9d20956cdebca04841316660ed584c2fecdc2b81722a28bc7ad3b1e4/tiktoken-0.13.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:4d9980f11429ed2d737c463bb1fb78cf330caa026adf002f714aced7849a687b", size = 982970, upload-time = "2026-05-15T04:50:32.961Z" }, + { url = "https://files.pythonhosted.org/packages/74/ed/6bb8d05b9f731f749fee5c6f5ca63e981143c826a5985877330507bd13b7/tiktoken-0.13.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:3f277ebea5edd7b8bf03c6f9431e1d67d517530115572b2dc1d465326e8f88c7", size = 1115741, upload-time = "2026-05-15T04:50:34.475Z" }, + { url = "https://files.pythonhosted.org/packages/34/de/2ca96b07a82d972b74fe4b46de055b79c904e45c7eab699354a0bfa697dc/tiktoken-0.13.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:a116178fa7e1b4065bff05214360373a65cac22f965be7b3f73d00a0dbfe7649", size = 1136523, upload-time = "2026-05-15T04:50:35.782Z" }, + { url = "https://files.pythonhosted.org/packages/ee/dc/9dafec002c2d4424378563cf4cf5c7fb93631d2a55013c8b87554ee4012c/tiktoken-0.13.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:2c397ddda233208345b01bd30f2fca79ff730e55731d0108a603f9bc57f6af3b", size = 1181954, upload-time = "2026-05-15T04:50:36.99Z" }, + { url = "https://files.pythonhosted.org/packages/a1/d0/1f8578c45b2f24759b46f0b50d31878c63c73e6bf0f2227e10ec5c5408dc/tiktoken-0.13.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:95097e4f89b06403976e498abf61a0ee73a7497e73fb599cb211d8197a054d91", size = 1240069, upload-time = "2026-05-15T04:50:38.221Z" }, + { url = "https://files.pythonhosted.org/packages/aa/90/28d7f154888610aa9237e541986beb62b479df29d193a5a0617dbb1514d0/tiktoken-0.13.0-cp312-cp312-win_amd64.whl", hash = "sha256:8f2d16e7a7c783ad81f36e457d046d1f1c8af70b22aec8a13238efe531977c41", size = 874748, upload-time = "2026-05-15T04:50:39.587Z" }, +] + +[[package]] +name = "tokenizers" +version = "0.23.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "huggingface-hub" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/c1/60/21f715d9faba5f5407ff759472ade058ec4a507ad62bcea47cb847239a73/tokenizers-0.23.1.tar.gz", hash = "sha256:1feeeadf865a7915adc25445dea30e9933e593c31bb96c277cee36de227c8bfa", size = 365748, upload-time = "2026-04-27T14:43:25.606Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/87/39/b87a87d5bb9470610b80a2d31df42fcffeaf35118b8b97952b2aff598cc7/tokenizers-0.23.1-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:e03d6ffcbe0d56ee9c1ccd070e70a13fa750727c0277e138152acbc0252c2224", size = 3146732, upload-time = "2026-04-27T14:43:15.427Z" }, + { url = "https://files.pythonhosted.org/packages/e2/6a/068ed9f6e444c9d7e9d55ce134181325700f3d7f30410721bdc8f848d727/tokenizers-0.23.1-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:e0948bbb1ac1d7cdfc9fb6d62c596e3b7550036ad60ecd654a66ad273326324e", size = 3054954, upload-time = "2026-04-27T14:43:13.745Z" }, + { url = "https://files.pythonhosted.org/packages/6c/36/e006edf031154cba92b8416057d92c3abe3635e4c4b0aa0b5b9bb39dde70/tokenizers-0.23.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1bf13402aff9bc533c89cb849ec3b412dc3fbeacc9744840e423d7bf3f7dc0e3", size = 3374081, upload-time = "2026-04-27T14:43:01.241Z" }, + { url = "https://files.pythonhosted.org/packages/a2/ef/7735d226f9c7f874a6bee5e3f27fb25ecabdf207d37b8cf45286d0795893/tokenizers-0.23.1-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:f836ca703b89ae07919a309f9651f7a88fd5a33d5f718ba5ad0870ec0256bad6", size = 3247641, upload-time = "2026-04-27T14:43:03.856Z" }, + { url = "https://files.pythonhosted.org/packages/b9/d9/24827036f6e21297bfffda0768e58eb6096a4f411e932964a01707857931/tokenizers-0.23.1-cp310-abi3-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:ae848657742035523fdf261773630cb819a26995fcd3d9ecae0c1daf6e5a4959", size = 3585624, upload-time = "2026-04-27T14:43:10.664Z" }, + { url = "https://files.pythonhosted.org/packages/0c/9a/22f3582b3a4f49358293a5206e25317621ee4526bfe9cdaa0f07a12e770e/tokenizers-0.23.1-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:53b09e85775d5187941e7bab30e941b4134ab4a7dd8c68e783d231fb7ca27c51", size = 3844062, upload-time = "2026-04-27T14:43:05.643Z" }, + { url = "https://files.pythonhosted.org/packages/7e/65/b8f8814eef95800f20721384136d9a1d22241d50b2874357cb70542c392f/tokenizers-0.23.1-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:ea5a0ce170074329faaa8ea3f6400ecde604b6678192688533af80980daae71a", size = 3460098, upload-time = "2026-04-27T14:43:08.854Z" }, + { url = "https://files.pythonhosted.org/packages/0d/d5/1353e5f677ec27c2494fb6a6725e82d56c985f53e90ec511369e7e4f02c6/tokenizers-0.23.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5075b405006415ea148a992d093699c66eb01952bf59f4d5727089a98bda45a4", size = 3346235, upload-time = "2026-04-27T14:43:12.377Z" }, + { url = "https://files.pythonhosted.org/packages/71/89/39b6b8fc073fb6d413d0147aa333dc7eff7be65639ac9d19930a0b21bf33/tokenizers-0.23.1-cp310-abi3-manylinux_2_31_riscv64.whl", hash = "sha256:56f3a77de629917652f876294dc9fe6bad4a0c43bc229dc72e59bb23a0f4729a", size = 3426398, upload-time = "2026-04-27T14:43:07.264Z" }, + { url = "https://files.pythonhosted.org/packages/0f/80/127c854da64827e5b79264ce524993a90dddcb320e5cd42412c5c02f9e8a/tokenizers-0.23.1-cp310-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:9d10a6d957ef01896dc274e890eee27d41bd0e74ef31e60616f0fc311345184e", size = 9823279, upload-time = "2026-04-27T14:43:17.222Z" }, + { url = "https://files.pythonhosted.org/packages/fe/ba/44c2502feb1a058f096ddfb4e0996ef3225a01a388e1a9b094e91689fe93/tokenizers-0.23.1-cp310-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:1974288a609c343774f1b897c8b482c791ab17b75ab5c8c2b1737565c1d82288", size = 9644986, upload-time = "2026-04-27T14:43:19.45Z" }, + { url = "https://files.pythonhosted.org/packages/9e/c1/464019a9fb059870bfe4eebb4ba12208f3042035e258bf5e782906bd3847/tokenizers-0.23.1-cp310-abi3-musllinux_1_2_i686.whl", hash = "sha256:120468fb4c24faf0543c835a4fabafa4deb3f20a035c9b6e83d0b553a97615d4", size = 9976181, upload-time = "2026-04-27T14:43:21.463Z" }, + { url = "https://files.pythonhosted.org/packages/79/94/3ac1432bda31626071e9b6a12709b97ae05131c804b94c8f3ac622c5da32/tokenizers-0.23.1-cp310-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:e3d8f40ea6268047de7046906326abed5134f27d4e8447b23763afe5808c8a96", size = 10113853, upload-time = "2026-04-27T14:43:23.617Z" }, + { url = "https://files.pythonhosted.org/packages/6a/dd/631b21433c771b1382535326f0eca80b9c9cee2e64961dd993bc9ac4669e/tokenizers-0.23.1-cp310-abi3-win32.whl", hash = "sha256:93120a930b919416da7cd10a2f606ac9919cc69cacae7980fa2140e277660948", size = 2536263, upload-time = "2026-04-27T14:43:29.888Z" }, + { url = "https://files.pythonhosted.org/packages/97/c9/2553f72aaf65a2797d4229e37fa7fbe38ffbf3e32912d31bdd78b3323e59/tokenizers-0.23.1-cp310-abi3-win_amd64.whl", hash = "sha256:e7bfaf995c1bdbbd21d13539decb6650967013759318627d85daeb7881af16b7", size = 2798223, upload-time = "2026-04-27T14:43:28.51Z" }, + { url = "https://files.pythonhosted.org/packages/cd/2b/2be299bab55fc595e3d38567edb1a87f86e594842968fa9515a07bdcf422/tokenizers-0.23.1-cp310-abi3-win_arm64.whl", hash = "sha256:a26197957d8e4425dfba746315f3c425ea00cfa8367c5fbc4ec73447893dcea9", size = 2664127, upload-time = "2026-04-27T14:43:26.949Z" }, +] + +[[package]] +name = "toml" +version = "0.10.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/be/ba/1f744cdc819428fc6b5084ec34d9b30660f6f9daaf70eead706e3203ec3c/toml-0.10.2.tar.gz", hash = "sha256:b3bda1d108d5dd99f4a20d24d9c348e91c4db7ab1b749200bded2f839ccbe68f", size = 22253, upload-time = "2020-11-01T01:40:22.204Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", hash = "sha256:806143ae5bfb6a3c6e736a764057db0e6a0e05e338b5630894a5f779cabb4f9b", size = 16588, upload-time = "2020-11-01T01:40:20.672Z" }, +] + +[[package]] +name = "tqdm" +version = "4.68.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/ae/5f/57ff8b434839e70dab45601284ea413e947a63799891b7553e5960a793a8/tqdm-4.68.4.tar.gz", hash = "sha256:19829c9673638f2a0b8617da4cdcb927e831cd88bcfcb6e78d42a4d1af131520", size = 792418, upload-time = "2026-07-07T09:58:18.369Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/22/2a/5e5e750890ada51017d18d0d4c30da696e5b5bd3180947729927628fc3cb/tqdm-4.68.4-py3-none-any.whl", hash = "sha256:5168118b2368f48c561afda8020fd79195b1bdb0bdf8086b88442c267a315dc2", size = 676612, upload-time = "2026-07-07T09:58:16.256Z" }, +] + +[[package]] +name = "typer" +version = "0.26.8" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "annotated-doc" }, + { name = "colorama", marker = "sys_platform == 'win32'" }, + { name = "rich" }, + { name = "shellingham" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/7c/f7/68adc395201b20b872d68e975386832e8005ffeacedd43a1d837a32815be/typer-0.26.8.tar.gz", hash = "sha256:c244a6bd558886fe3f8780efb6bdd28bb9aff005a94eedebaa5cb32926fe2f7e", size = 202097, upload-time = "2026-06-26T09:22:45.705Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/80/87/b9fd69c92c6102a066e1b86a35243f53e70bd4c709f2a26d9f4fee4f4dc0/typer-0.26.8-py3-none-any.whl", hash = "sha256:3512ca79ac5c11113414b36e80281b872884477722440691c89d1112e321a49c", size = 122564, upload-time = "2026-06-26T09:22:44.72Z" }, +] + +[[package]] +name = "types-certifi" +version = "2021.10.8.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/52/68/943c3aeaf14624712a0357c4a67814dba5cea36d194f5c764dad7959a00c/types-certifi-2021.10.8.3.tar.gz", hash = "sha256:72cf7798d165bc0b76e1c10dd1ea3097c7063c42c21d664523b928e88b554a4f", size = 2095, upload-time = "2022-06-09T15:19:05.244Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b5/63/2463d89481e811f007b0e1cd0a91e52e141b47f9de724d20db7b861dcfec/types_certifi-2021.10.8.3-py3-none-any.whl", hash = "sha256:b2d1e325e69f71f7c78e5943d410e650b4707bb0ef32e4ddf3da37f54176e88a", size = 2136, upload-time = "2022-06-09T15:19:03.127Z" }, +] + +[[package]] +name = "types-toml" +version = "0.10.8.20260518" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/4b/11/6ece999e91f2ccb848ab4420f3f4816e78ac0541f739e6864affdaaa5737/types_toml-0.10.8.20260518.tar.gz", hash = "sha256:80e10facd24fdeda9d5c672187d72be3ac284843788d67f5aae59e3e016db6fe", size = 9419, upload-time = "2026-05-18T06:02:16.719Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/91/25/489751806bf5c95e4007f8e17409199c54d31e49ffbea07c5729b1286c8e/types_toml-0.10.8.20260518-py3-none-any.whl", hash = "sha256:0e564ab05f6fde62a315b3b5a9b6624fda569399795d30a37e64705a70459303", size = 9669, upload-time = "2026-05-18T06:02:15.86Z" }, +] + +[[package]] +name = "typing-extensions" +version = "4.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f6/cc/6253133b5bb138fc3306cebfbda2c520f545d36b5be2c7255cc528bb45d6/typing_extensions-4.16.0.tar.gz", hash = "sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5", size = 113555, upload-time = "2026-07-02T08:40:05.92Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/49/d3/b8441a820a491ddfc024b0b0cf0393375b75ea13866d9c66727e54c2fc80/typing_extensions-4.16.0-py3-none-any.whl", hash = "sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8", size = 45571, upload-time = "2026-07-02T08:40:04.659Z" }, +] + +[[package]] +name = "typing-inspection" +version = "0.4.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/55/e3/70399cb7dd41c10ac53367ae42139cf4b1ca5f36bb3dc6c9d33acdb43655/typing_inspection-0.4.2.tar.gz", hash = "sha256:ba561c48a67c5958007083d386c3295464928b01faa735ab8547c5692e87f464", size = 75949, upload-time = "2025-10-01T02:14:41.687Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl", hash = "sha256:4ed1cacbdc298c220f1bd249ed5287caa16f34d44ef4e9c3d0cbad5b521545e7", size = 14611, upload-time = "2025-10-01T02:14:40.154Z" }, +] + +[[package]] +name = "tzdata" +version = "2026.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/92/ff/5a28bdfd8c3ebec42564ac7d0e54ca3db65044a9314a97f9564fa7a1e926/tzdata-2026.3.tar.gz", hash = "sha256:4a1518b8993086a7982523e071643f3c0e5f213e75b21318e78bcabfff9d1415", size = 198674, upload-time = "2026-07-10T08:50:37.887Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e5/6d/b53b99a9f2766d095985947a5782f1702cabb129a34f7a802d7197af832f/tzdata-2026.3-py2.py3-none-any.whl", hash = "sha256:dc096730c87af6cab1b171c9d532be840741ff5d459015e7f6947bd7d7e54931", size = 348168, upload-time = "2026-07-10T08:50:36.46Z" }, +] + +[[package]] +name = "uc-micro-py" +version = "2.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/78/67/9a363818028526e2d4579334460df777115bdec1bb77c08f9db88f6389f2/uc_micro_py-2.0.0.tar.gz", hash = "sha256:c53691e495c8db60e16ffc4861a35469b0ba0821fe409a8a7a0a71864d33a811", size = 6611, upload-time = "2026-03-01T06:31:27.526Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/61/73/d21edf5b204d1467e06500080a50f79d49ef2b997c79123a536d4a17d97c/uc_micro_py-2.0.0-py3-none-any.whl", hash = "sha256:3603a3859af53e5a39bc7677713c78ea6589ff188d70f4fee165db88e22b242c", size = 6383, upload-time = "2026-03-01T06:31:26.257Z" }, +] + +[[package]] +name = "unidiff" +version = "0.7.5" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a3/48/81be0ac96e423a877754153699731ef439fd7b80b4c8b5425c94ed079ebd/unidiff-0.7.5.tar.gz", hash = "sha256:2e5f0162052248946b9f0970a40e9e124236bf86c82b70821143a6fc1dea2574", size = 20931, upload-time = "2023-03-10T01:05:39.185Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/8a/54/57c411a6e8f7bd7848c8b66e4dcaffa586bf4c02e63f2280db0327a4e6eb/unidiff-0.7.5-py2.py3-none-any.whl", hash = "sha256:c93bf2265cc1ba2a520e415ab05da587370bc2a3ae9e0414329f54f0c2fc09e8", size = 14386, upload-time = "2023-03-10T01:05:36.594Z" }, +] + +[[package]] +name = "urllib3" +version = "2.7.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/53/0c/06f8b233b8fd13b9e5ee11424ef85419ba0d8ba0b3138bf360be2ff56953/urllib3-2.7.0.tar.gz", hash = "sha256:231e0ec3b63ceb14667c67be60f2f2c40a518cb38b03af60abc813da26505f4c", size = 433602, upload-time = "2026-05-07T16:13:18.596Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl", hash = "sha256:9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897", size = 131087, upload-time = "2026-05-07T16:13:17.151Z" }, +] + +[[package]] +name = "virtualenv" +version = "21.6.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "distlib" }, + { name = "filelock" }, + { name = "platformdirs" }, + { name = "python-discovery" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/34/d9/b477fddb68840b570af8b22afe9b035cbc277b5fb7b33dea390617a8b10f/virtualenv-21.6.1.tar.gz", hash = "sha256:15f978b7cd329f24855ff4a0c4b4899cc7678589f49adbdcbbb4d3232e641128", size = 5526620, upload-time = "2026-07-10T19:33:53.312Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c1/7c/4e7225d46d634a0d8d534dd8a6ce0c319d09b4d0cf0337eb314ca4789d8c/virtualenv-21.6.1-py3-none-any.whl", hash = "sha256:afe991df855715a2b2f60edfcc0107ef95a79fdfd8cb4cdaa71603d1c12e463b", size = 5506392, upload-time = "2026-07-10T19:33:51.629Z" }, +] + +[[package]] +name = "watchfiles" +version = "1.2.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/cd/41/5e1a4bb12aac5f1493fa1bdc11154eca3b258ca4eba65d39c473fe19d8e9/watchfiles-1.2.0.tar.gz", hash = "sha256:c995fba777f1ea992f090f9236e9284cf7a5d1a0130dd5a3d82c598cacd76838", size = 108252, upload-time = "2026-05-18T04:32:04.251Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b8/2f/e42c992d2afda3108ea1c02acecc991b9f31d05c14adc2a7cee9ee211fc4/watchfiles-1.2.0-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:bc13eb17538be00c874699dc0abe4ee2bc8d50bb1166a6b9e175ef3fd7eb8f26", size = 400115, upload-time = "2026-05-18T04:32:02.06Z" }, + { url = "https://files.pythonhosted.org/packages/5f/8f/6af2ea19065c91d8b0ea3516fdfc8c0d349f407e8e9fbf4e5a17360de8ad/watchfiles-1.2.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:2d95ddc1eb6914154253d239089900813f6a767e174b8e6a50e7fdacb7e4236c", size = 393659, upload-time = "2026-05-18T04:30:50.951Z" }, + { url = "https://files.pythonhosted.org/packages/13/01/b32a967c56fb3e3e5be3db52c3d3b87fa4513aa367d8ed1ad96d42952e5f/watchfiles-1.2.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:8f70d8b291ef6e88d19b1f297a6905ddb978888d9272b0d05e6f53309856bcfc", size = 453207, upload-time = "2026-05-18T04:31:04.231Z" }, + { url = "https://files.pythonhosted.org/packages/04/98/97557a812180338cb1abd32e1cffcc4588f59b5f23e0cb006b2ba95ba64a/watchfiles-1.2.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:56d8641cf834c2836922899105bd3ce3d0dfc69291d52edf0b4d0436829b34c0", size = 459273, upload-time = "2026-05-18T04:31:50.377Z" }, + { url = "https://files.pythonhosted.org/packages/e8/a8/b4b08dcb7653b8087c6586f7ce649505900e866bbcfe40dc9587af02e686/watchfiles-1.2.0-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:2581a94056e55d7d0a31a823ea92bf73749c489ca2285bfdc0fbe6b2bb49d50c", size = 489927, upload-time = "2026-05-18T04:31:42.485Z" }, + { url = "https://files.pythonhosted.org/packages/50/94/3dceea03545d2e5ddfd839f0ddd5e1cecbf1697b5a428d5ba11cef6af95d/watchfiles-1.2.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:41bc1199f7523b3f82843c88cbb979180c949caef0342cf90968f178e5d49b01", size = 570476, upload-time = "2026-05-18T04:31:03.071Z" }, + { url = "https://files.pythonhosted.org/packages/cc/f2/d39a5450c3532092b91f81d274360e613c2371bc874a89c7a1a3c5e8d138/watchfiles-1.2.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:7571e4464cb6e434958f867f7f730b8ab0b75e3f8e5eac0499168486ab3c33a8", size = 465650, upload-time = "2026-05-18T04:30:12.701Z" }, + { url = "https://files.pythonhosted.org/packages/22/24/ed72f68cbc1333ca9b9f2200aa048bb6658ae41709bc1caad4310f4bdffd/watchfiles-1.2.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e53a384f76b631c3ae5334ce6a52f0baa3a911eb94a4eac7f160079868b716d5", size = 456398, upload-time = "2026-05-18T04:30:13.784Z" }, + { url = "https://files.pythonhosted.org/packages/0d/64/982ef4a4e5bab5b6e5b6becc8cd5e732f6130a78b855f0abec6439a9a135/watchfiles-1.2.0-cp312-cp312-manylinux_2_31_riscv64.whl", hash = "sha256:d20029a60a71a052a24c4db7673bc4de39ab89adbaccbfb5d67987c5d73f424d", size = 465140, upload-time = "2026-05-18T04:31:52.111Z" }, + { url = "https://files.pythonhosted.org/packages/a0/0c/95282abf4ed680b6096010bcfc30c5fa7a041fc5aa5a2ad17a2cc6c75bba/watchfiles-1.2.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:2cb93af48550faf1cea04c303107c8b75833de7013e57ce27d3b8d21d8d0f58c", size = 630259, upload-time = "2026-05-18T04:31:25.676Z" }, + { url = "https://files.pythonhosted.org/packages/30/45/607c1de1530c4bdcf2cf1d1ecc2505ddba5d96bd43ba9f2b0e79876f850f/watchfiles-1.2.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:2995c176de7692b86a2e4c58d9ec718f753150a979cb4a754e2b4ffa38e70906", size = 659859, upload-time = "2026-05-18T04:30:24.333Z" }, + { url = "https://files.pythonhosted.org/packages/fa/08/d9e2e0f9e8e6791d33aefc694ad7eefa7f901f63caff84a81ded38692f9c/watchfiles-1.2.0-cp312-cp312-win32.whl", hash = "sha256:7a2cffd17d27d2ecbb310c2b1d8174f222a5495b1a721894afa88ec11e25b898", size = 275480, upload-time = "2026-05-18T04:30:31.307Z" }, + { url = "https://files.pythonhosted.org/packages/1c/e6/9d42569c0102645cc8cea5d8c7d8a1e9d4ada2cb7f05f75e554b8aa2202a/watchfiles-1.2.0-cp312-cp312-win_amd64.whl", hash = "sha256:f155b3a1b2a5fc89cdc70d47ee5d54e3b75e88efa34982028a35daef9ba00379", size = 288718, upload-time = "2026-05-18T04:32:10.745Z" }, + { url = "https://files.pythonhosted.org/packages/0a/26/88e0dc6ee3898169d7fa22bb6a69cabf2502d2ee25cb8c876d1262d204f8/watchfiles-1.2.0-cp312-cp312-win_arm64.whl", hash = "sha256:8fa585ede612ee9f9e91b18bebf9ba11b9ae29a4e3a0d0cf6fca3e382133f0d5", size = 281026, upload-time = "2026-05-18T04:30:22.23Z" }, +] + +[[package]] +name = "wcwidth" +version = "0.8.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/34/74/c6428f875774288bec1396f5bfcbc2d925700a4dad61727fd5f2b12f249d/wcwidth-0.8.2.tar.gz", hash = "sha256:91fbef97204b96a3d4d421609b80340b760cf33e26da123ff243d76b1fda8dda", size = 1466253, upload-time = "2026-06-29T18:11:11.601Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/96/42/3e5985a0a7e57de470b320c6d6a1a67c844f6737a587f3d44dd13d1819e7/wcwidth-0.8.2-py3-none-any.whl", hash = "sha256:d63947694a0539a1d51e01eda7caf800c291020e6cdd7e28ad7b14dd33ad4f85", size = 323166, upload-time = "2026-06-29T18:11:09.888Z" }, +] + +[[package]] +name = "xxhash" +version = "3.8.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/8e/63/71aa56b151a1b28770037a61bd4e461c2619cfc8866a4fcaf1548605e325/xxhash-3.8.1.tar.gz", hash = "sha256:b0de4bf3aa66363552d52c6a89003c479911f12098cd48a53d44a0f7a25f7c46", size = 86223, upload-time = "2026-07-06T10:49:58.937Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/42/91/f65c34a7aa7b4e7cf4854f8e6ef3f7ee32ceac41d4f008da0780db0612f6/xxhash-3.8.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:e6e49370822c1f4d8d90e678b06dbcb08b51a026a7c4b55479e7d467f2e813bc", size = 34680, upload-time = "2026-07-06T10:44:40.932Z" }, + { url = "https://files.pythonhosted.org/packages/57/04/b10a245a4c09a9cfa88f8e9ae755029413ad1ac17047f9a61906e5ae0799/xxhash-3.8.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:220d68130f83f7cc86d6edfdeab176adc73d7200bf3a8ec10c629e8cf605c215", size = 32397, upload-time = "2026-07-06T10:44:42.196Z" }, + { url = "https://files.pythonhosted.org/packages/3a/75/45ab795b5945b6388583bd75202106af505537935566c15a1577797a0e08/xxhash-3.8.1-cp312-cp312-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:4d365ee1892c1fa803536f8c6ce21d24b29c9718ec75eb856095c07830f8c478", size = 220549, upload-time = "2026-07-06T10:44:43.603Z" }, + { url = "https://files.pythonhosted.org/packages/13/44/5ba2bd0a14ddf4193fc7d8ec29625f659f22c06d60b28f04bf46305d8330/xxhash-3.8.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:852bfe059720632e2f16a6a4745e41d20937b2bf2a42a401e2412046bb6971cc", size = 241186, upload-time = "2026-07-06T10:44:45.534Z" }, + { url = "https://files.pythonhosted.org/packages/23/32/c4147def4d1e4538b906f82731e0ba23424377fc50a7cddd03cd284c8f63/xxhash-3.8.1-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:2f8c25a7061d952de589bd0ea0eaadee32378ff83dd6a677b267f9cd86f401f8", size = 264852, upload-time = "2026-07-06T10:44:47.199Z" }, + { url = "https://files.pythonhosted.org/packages/6c/bd/71ed14f4f0318bb7fd7b2ec51999413487fa8da8d41208e84d50d1ef0f98/xxhash-3.8.1-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:868a8dcaff1a84ba78038e1cef14fc88ccf84d9b4d12ea604696e0693296aa56", size = 242663, upload-time = "2026-07-06T10:44:48.846Z" }, + { url = "https://files.pythonhosted.org/packages/91/09/70af22c565a8473b3f2ae73f88e7721af281bc4a575236dbd1970c9f76f6/xxhash-3.8.1-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:6536d8677d2fff7e64cd0b98b976df9de7aee0e69590044c2af5f51b76b7a170", size = 473510, upload-time = "2026-07-06T10:44:50.695Z" }, + { url = "https://files.pythonhosted.org/packages/18/96/34db781c8f0cf99c544ca1f2bc2e5bf55426e1eb4ca6de8ea5da56a9f352/xxhash-3.8.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:82c0cedd280eab2e8291270e6c04894dbc096f8159a39dcf1807429f026ca3cc", size = 220469, upload-time = "2026-07-06T10:44:52.422Z" }, + { url = "https://files.pythonhosted.org/packages/93/5f/9a184f615fa5a4dce30c01534f62946ce5a11ce40f73785cbd356ccabaa9/xxhash-3.8.1-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:daa86e4b68221d38e669bb236ba112d0335353829fb627c82e5909e4bbe8694c", size = 310290, upload-time = "2026-07-06T10:44:54.142Z" }, + { url = "https://files.pythonhosted.org/packages/a9/dc/9b9a9789011ee153723a5eb9e7dd7fcbae2ba9b3fe7a729249ca7c252056/xxhash-3.8.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:2bc7113e6f2b6b3922dd61796ca9f36af09da3773898e7003038dc992fc83b8d", size = 238173, upload-time = "2026-07-06T10:44:55.693Z" }, + { url = "https://files.pythonhosted.org/packages/ec/4d/71c6005ada9dcb608a4e1902e8475ecadb5f3fbfa04e1e244d276a2d0c43/xxhash-3.8.1-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:5eed32dad81d6ba8e62dc7b9ffa0500199385d7810a8dd9d4eafaceb8c6e20bb", size = 269026, upload-time = "2026-07-06T10:44:57.424Z" }, + { url = "https://files.pythonhosted.org/packages/2f/87/d6c036ba25dfbd9c8633be5aa86fc9474bbb9e2c68212a841d090abe7344/xxhash-3.8.1-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:83697b0ea1f10e7f5d8b26a4906fa851393c61546c63839643a2b7fe2d868061", size = 224970, upload-time = "2026-07-06T10:44:59.085Z" }, + { url = "https://files.pythonhosted.org/packages/48/62/4c1f035a41c5752aa05e195b6c904c07b94fe9061a16de61e72a6e6b135f/xxhash-3.8.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:36fc69160465ae75c6ec4ac9f781bb2aa16ae7ff869e73c26fee85fbb11b9887", size = 240820, upload-time = "2026-07-06T10:45:00.746Z" }, + { url = "https://files.pythonhosted.org/packages/da/14/d39d565069b87e86d21a2af2a31d04db79249d25aa8d5b62959056a89857/xxhash-3.8.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:445e0f5a31f2f3546ae0895d4811e159518cdc9d824c11419898d40cfadb677e", size = 300619, upload-time = "2026-07-06T10:45:02.716Z" }, + { url = "https://files.pythonhosted.org/packages/13/22/75467acc887edc8cf71c97ab1708feb3df7a88bda589b9f399765c6387d2/xxhash-3.8.1-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:dfe0580fbfd5e4af87d0cc52d2044f155d55ebd8c8a93568758a2ea7d8e15975", size = 443267, upload-time = "2026-07-06T10:45:04.653Z" }, + { url = "https://files.pythonhosted.org/packages/a4/b6/1da3baa5fa6ef705e3425fddd382be7dfc4dfba2686df90a20f16e9c7b1b/xxhash-3.8.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:095e1323fa108be1292c54c86da3ef3c7a7dc015b105a52133973bc07a6ad11a", size = 217338, upload-time = "2026-07-06T10:45:06.304Z" }, + { url = "https://files.pythonhosted.org/packages/78/dd/b5295a9f97484e7a1c2b283a742ca45e3104991c55a1ef670dde161829ba/xxhash-3.8.1-cp312-cp312-win32.whl", hash = "sha256:bf28f55e427e0483acb1f666bd0d869b6d5e5a716680c216ad7befe3d4cfba2e", size = 31970, upload-time = "2026-07-06T10:45:07.823Z" }, + { url = "https://files.pythonhosted.org/packages/ec/31/3fa0b807d7e21515cd975e7fe5c039d52ac3e9401a96d6ad68dae6305215/xxhash-3.8.1-cp312-cp312-win_amd64.whl", hash = "sha256:2256e80e4960ee282f63428adb349cb7f8bd8efe4db770d88eb815f4b9860724", size = 32741, upload-time = "2026-07-06T10:45:09.42Z" }, + { url = "https://files.pythonhosted.org/packages/b8/05/86feada74e239600e6875aa507afb40482a89b92700aa74a92da83bdcb77/xxhash-3.8.1-cp312-cp312-win_arm64.whl", hash = "sha256:9df56e6df96a60590935e22373041cccc91fd55858763dcffb55bf63b3a2b396", size = 29234, upload-time = "2026-07-06T10:45:10.809Z" }, +] + +[[package]] +name = "yarl" +version = "1.24.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "idna" }, + { name = "multidict" }, + { name = "propcache" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/79/12/1e8f37460ea0f7eb59c221fdaf0ed75e7ac43e97f8093b9c6f411df50a78/yarl-1.24.2.tar.gz", hash = "sha256:9ac374123c6fd7abf64d1fec93962b0bd4ee2c19751755a762a72dd96c0378f8", size = 210798, upload-time = "2026-05-19T21:31:05.599Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f0/da/866bcb01076ba49d2b42b309867bed3826421f1c479655eb7a607b44f20b/yarl-1.24.2-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:b975866c184564c827e0877380f0dae57dcca7e52782128381b72feff6dfceb8", size = 129957, upload-time = "2026-05-19T21:28:51.695Z" }, + { url = "https://files.pythonhosted.org/packages/bf/1d/fcefb70922ea2268a8971d8e5874d9a8218644200fb8465f1dcad55e6851/yarl-1.24.2-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:3b075301a2836a0e297b1b658cb6d6135df535d62efefdd60366bd589c2c82f2", size = 92164, upload-time = "2026-05-19T21:28:53.242Z" }, + { url = "https://files.pythonhosted.org/packages/29/b6/170e2b8d4e3bc30e6bfdcca53556537f5bf595e938632dfcb059311f3ff6/yarl-1.24.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:8ae44649b00947634ab0dab2a374a638f52923a6e67083f2c156cd5cbd1a881d", size = 91688, upload-time = "2026-05-19T21:28:54.865Z" }, + { url = "https://files.pythonhosted.org/packages/fe/a5/c9f655d5553ea0b99fdac9d6a99ad3f9b3e73b8e5758bb46f58c9831f74c/yarl-1.24.2-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:507cc19f0b45454e2d6dcd62ff7d062b9f77a2812404e62dbdaec05b50faa035", size = 102902, upload-time = "2026-05-19T21:28:56.963Z" }, + { url = "https://files.pythonhosted.org/packages/5d/bc/6b9664d815d79af4ee553337f9d606c56bbf269186ada9172de45f1b5f60/yarl-1.24.2-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:c4c17bad5a530912d2111825d3f05e89bab2dd376aaa8cbc77e449e6db63e576", size = 97931, upload-time = "2026-05-19T21:28:58.56Z" }, + { url = "https://files.pythonhosted.org/packages/98/ec/32ba48acae30fecd60928f5791188b80a9d6ee3840507ffda29fecd37b71/yarl-1.24.2-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f5f0cbb112838a4a293985b6ed73948a547dadcc1ba6d2089938e7abdedceef8", size = 111030, upload-time = "2026-05-19T21:29:00.148Z" }, + { url = "https://files.pythonhosted.org/packages/82/5a/6f4cd081e5f4934d2ae3a8ef4abe3afacc010d26f0035ee91b35cd7d7c37/yarl-1.24.2-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5ec8356b8a6afcf81fc7aeeef13b1ff7a49dec00f313394bbb9e83830d32ccd7", size = 110392, upload-time = "2026-05-19T21:29:02.155Z" }, + { url = "https://files.pythonhosted.org/packages/7a/da/323a01c349bd5fb01bb6652e314d9bb218cee630a736bdb810ad50e4013f/yarl-1.24.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7e7ebcdef69dec6c6451e616f32b622a6d4a2e92b445c992f7c8e5274a6bbc4c", size = 105612, upload-time = "2026-05-19T21:29:04.247Z" }, + { url = "https://files.pythonhosted.org/packages/7c/80/264ab684f181e1a876389374519ff05d10248725535ae2ac4e8ac4e563d6/yarl-1.24.2-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:47a55d6cf6db2f401017a9e96e5288844e5051911fb4e0c8311a3980f5e59a7d", size = 104487, upload-time = "2026-05-19T21:29:06.491Z" }, + { url = "https://files.pythonhosted.org/packages/41/07/efabe5df87e96d7ad5959760b888344be48cd6884db127b407c6b5503adc/yarl-1.24.2-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:3065657c80a2321225e804048597ad55658a7e76b32d6f5ee4074d04c50401db", size = 102333, upload-time = "2026-05-19T21:29:08.267Z" }, + { url = "https://files.pythonhosted.org/packages/44/0c/bcf7c42603e1009295f586d8890f2ba032c8b53310e815adf0a202c73d9f/yarl-1.24.2-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:cb84b80d88e19ede158619b80813968713d8d008b0e2497a576e6a0557d50712", size = 99025, upload-time = "2026-05-19T21:29:10.682Z" }, + { url = "https://files.pythonhosted.org/packages/4f/82/84482ab1a57a0f21a08afe6a7004c61d741f8f2ecc3b05c321577c612164/yarl-1.24.2-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:990de4f680b1c217e77ff0d6aa0029f9eb79889c11fb3e9a3942c7eba29c1996", size = 110507, upload-time = "2026-05-19T21:29:12.954Z" }, + { url = "https://files.pythonhosted.org/packages/c4/8d/a546ba1dfe1b0f290e05fef145cd07614c0f15df1a707195e512d1e39d1d/yarl-1.24.2-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:abb8ec0323b80161e3802da3150ef660b41d0e9be2048b76a363d93eee992c2b", size = 103719, upload-time = "2026-05-19T21:29:14.893Z" }, + { url = "https://files.pythonhosted.org/packages/1a/b6/267f2a09213138473adfce6b8a6e17791d7fee70bd4d9003218e4dec58b0/yarl-1.24.2-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:e7977781f83638a4c73e0f88425563d70173e0dfd90ac006a45c65036293ee3c", size = 110438, upload-time = "2026-05-19T21:29:16.485Z" }, + { url = "https://files.pythonhosted.org/packages/48/2d/1c8d89c7c5f9cad9fb2902445d94e2ab1d7aa35de029afbb8ae95c42d00f/yarl-1.24.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:e30dd55825dc554ec5b66a94953b8eda8745926514c5089dfcacecb9c99b5bd1", size = 105719, upload-time = "2026-05-19T21:29:18.367Z" }, + { url = "https://files.pythonhosted.org/packages/a7/25/722e3b93bd687009afb2d59a35e13d30ddd8f80571445bb0c4e4ce26ec66/yarl-1.24.2-cp312-cp312-win_amd64.whl", hash = "sha256:7dafe10c12ddd4d120d528c4b5599c953bd7b12845347d507b95451195bb6cad", size = 92901, upload-time = "2026-05-19T21:29:20.014Z" }, + { url = "https://files.pythonhosted.org/packages/39/47/4486ccfb674c04854a1ef8aa77868b6a6f765feaf69633409d7ca4f02cb8/yarl-1.24.2-cp312-cp312-win_arm64.whl", hash = "sha256:044a09d8401fcf8681977faef6d286b8ade1e2d2e9dceda175d1cfa5ca496f30", size = 87229, upload-time = "2026-05-19T21:29:22.1Z" }, + { url = "https://files.pythonhosted.org/packages/fd/4d/4b880086bd0d3e034d25647be1d830afc3e3f610e98c4ab3490af6b1b6d5/yarl-1.24.2-py3-none-any.whl", hash = "sha256:2783d9226db8797636cd6896e4de81feed252d1db72265686c9558d97a4d94b9", size = 53576, upload-time = "2026-05-19T21:31:03.909Z" }, +] + +[[package]] +name = "zipp" +version = "4.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/b9/d8/eab98a517c14134c0b2eb4e2387bc5f457334293ec5d2dd3857ec2966802/zipp-4.1.0.tar.gz", hash = "sha256:4cb57381f544315db7688e976e922a2b18cdb513d21cc194eb42232ba2a3e602", size = 26214, upload-time = "2026-05-18T20:08:57.967Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3a/13/547360d81e6d88d58492968ffda9f9542854f11310ee556fef14260cc886/zipp-4.1.0-py3-none-any.whl", hash = "sha256:25ad4e16390cd314347dd8f1de67a2ac538ae658ed4ab9db16029c07c188e97f", size = 10238, upload-time = "2026-05-18T20:08:57.045Z" }, +] diff --git a/tests/unit/commands/test_benchmark.py b/tests/unit/commands/test_benchmark.py index 491dc72b6..cc7287deb 100644 --- a/tests/unit/commands/test_benchmark.py +++ b/tests/unit/commands/test_benchmark.py @@ -27,6 +27,7 @@ from unittest.mock import MagicMock, patch from urllib import error as urllib_error +import inference_endpoint.commands.benchmark.execute as execute_mod import pandas as pd import pytest from inference_endpoint.commands.benchmark.cli import ( @@ -38,6 +39,7 @@ from inference_endpoint.commands.benchmark.execute import ( AccuracyConfiguration, BenchmarkContext, + BenchmarkResult, ResponseCollector, _build_phases, _derive_profile_urls, @@ -47,6 +49,7 @@ _render_profile_status, _run_benchmark_async, _write_profiling_section, + finalize_benchmark, setup_benchmark, ) from inference_endpoint.config.runtime_settings import RuntimeSettings @@ -75,11 +78,16 @@ from inference_endpoint.config.utils import cli_error_formatter as _error_formatter from inference_endpoint.core.types import QueryResult from inference_endpoint.dataset_manager.dataset import Dataset +from inference_endpoint.dataset_manager.predefined.swe_bench import SWEBench from inference_endpoint.endpoint_client.config import HTTPClientConfig -from inference_endpoint.evaluation.scoring import Scorer +from inference_endpoint.evaluation.scoring import Scorer, SWEBenchScorer from inference_endpoint.exceptions import InputValidationError, SetupError from inference_endpoint.load_generator.sample_order import create_sample_order -from inference_endpoint.load_generator.session import PhaseType +from inference_endpoint.load_generator.session import ( + PhaseResult, + PhaseType, + SessionResult, +) from inference_endpoint.metrics.metric import Throughput from pydantic import ValidationError @@ -91,6 +99,51 @@ / "templates" ) + +# Test-only scorers registered with leading-underscore IDs so TestScorerMethodSync excludes them. + + +class _SelfContainedScorer(Scorer, scorer_id="_test_skip_endpoint_phase"): + SKIP_ENDPOINT_PHASE = True + + def score_single_sample(self, value, ground_truth): + return 0.0 + + def score(self): + return 1.0, 1 + + +class _ExternalCountScorer(Scorer, scorer_id="_test_external_sample_count"): + SKIP_ENDPOINT_PHASE = True + + @classmethod + def external_sample_count(cls, extras): + return 2 + + def score_single_sample(self, value, ground_truth): + return 0.0 + + def score(self): + return 1.0, 1 + + +class _FailingPreflightScorer(Scorer, scorer_id="_test_failing_preflight"): + @classmethod + def preflight(cls, extras): + raise SetupError("mock preflight failure") + + def score_single_sample(self, value, ground_truth): + return 0.0 + + +class _ScorerShouldNotRun(Scorer, scorer_id="_test_scorer_should_not_run"): + def __init__(self, *args, **kwargs): + raise AssertionError("scorer should not be constructed") + + def score_single_sample(self, value, ground_truth): + return 0.0 + + # Reusable minimal config kwargs _OFFLINE_KWARGS = { "endpoint_config": {"endpoints": ["http://test:8000"]}, @@ -99,6 +152,70 @@ } +def _make_loaded_dataset(n: int = 3, *, column: str = "prompt") -> Dataset: + ds = Dataset(pd.DataFrame({column: [f"q{i}" for i in range(n)]})) + ds.load() + return ds + + +def _make_benchmark_context( + config: BenchmarkConfig, + report_dir: Path, + *, + test_mode: TestMode = TestMode.PERF, + dataloader: Dataset | None = None, + rt_settings: RuntimeSettings | None = None, + eval_configs: list[AccuracyConfiguration] | None = None, +) -> BenchmarkContext: + dataloader = dataloader or _make_loaded_dataset() + rt_settings = rt_settings or RuntimeSettings( + metric_target=Throughput(10.0), + reported_metrics=[Throughput(10.0)], + min_duration_ms=0, + max_duration_ms=None, + n_samples_from_dataset=dataloader.num_samples(), + n_samples_to_issue=None, + min_sample_count=1, + rng_sched=random.Random(0), + rng_sample_index=random.Random(0), + load_pattern=LoadPattern(type=LoadPatternType.MAX_THROUGHPUT), + ) + return BenchmarkContext( + config=config, + test_mode=test_mode, + report_dir=report_dir, + tokenizer_name=None, + dataloader=dataloader, + rt_settings=rt_settings, + total_samples=dataloader.num_samples(), + eval_configs=eval_configs or [], + ) + + +def _make_benchmark_result( + report_dir: Path, phase: PhaseResult | None = None +) -> BenchmarkResult: + phase = phase or PhaseResult( + name="performance", + phase_type=PhaseType.PERFORMANCE, + uuid_to_index={"uuid-0": 0, "uuid-1": 1, "uuid-2": 2}, + issued_count=3, + start_time_ns=0, + end_time_ns=1_000_000_000, + ) + return BenchmarkResult( + session=SessionResult( + session_id="test", + phase_results=[phase], + start_time_ns=0, + end_time_ns=1_000_000_000, + ), + collector=ResponseCollector(), + report=None, + tmpfs_dir=report_dir / "tmpfs", + ) + + class TestCLIConfigModels: """Test OfflineBenchmarkConfig/OnlineBenchmarkConfig defaults and validation.""" @@ -145,6 +262,51 @@ def test_missing_model_name_raises(self): datasets=[{"path": "test.jsonl"}], ) + @pytest.mark.unit + @pytest.mark.parametrize( + "accuracy_config, expected_workers", + [ + ({"eval_method": "swe_bench_scorer"}, 32), + ( + { + "eval_method": "swe_bench_scorer", + "extras": {"workers": None}, + }, + 32, + ), + ( + { + "eval_method": "swe_bench_scorer", + "extras": {"workers": 5}, + }, + 5, + ), + ], + ) + def test_concurrency_injection_into_swe_bench_extras( + self, accuracy_config, expected_workers + ): + """target_concurrency is forwarded as workers into swe_bench_scorer extras.""" + config = OnlineConfig( + endpoint_config={"endpoints": ["http://test:8000"]}, + model_params={"name": "test-model"}, + datasets=[ + { + "name": "swe_bench", + "type": "accuracy", + "accuracy_config": accuracy_config, + }, + {"type": "performance", "path": "tests/assets/datasets/dummy_1k.jsonl"}, + ], + settings={ + "load_pattern": {"type": "concurrency", "target_concurrency": 32} + }, + ) + acc_ds = next(d for d in config.datasets if d.type == DatasetType.ACCURACY) + assert acc_ds.accuracy_config is not None + assert acc_ds.accuracy_config.extras is not None + assert acc_ds.accuracy_config.extras.get("workers") == expected_workers + class TestDurationSuffix: """Test duration suffix parsing (600s, 10m, 600000ms, plain int).""" @@ -456,6 +618,246 @@ def test_validation_errors(self, overrides, match): ) +class TestAccuracyOnlyDataset: + """Test that datasets with ACCURACY_ONLY=True are rejected as perf datasets.""" + + @pytest.mark.unit + @pytest.mark.parametrize("dataset_name", ["swe_bench", "swe_bench::verified"]) + def test_swe_bench_as_perf_raises(self, tmp_path, dataset_name): + fake_df = pd.DataFrame( + [{"instance_id": "repo__repo-0", "problem_statement": "Fix bug 0"}] + ) + config = OfflineConfig( + endpoint_config={"endpoints": ["http://test:8000"]}, + model_params={"name": "test-model"}, + datasets=[{"name": dataset_name}], + ) + with ( + patch.object(SWEBench, "generate", return_value=fake_df), + pytest.raises(InputValidationError, match="accuracy-only"), + ): + _load_datasets(config, tmp_path, TestMode.PERF) + + @pytest.mark.unit + def test_preflight_error_propagates(self, tmp_path): + """A scorer whose preflight() raises SetupError must stop _load_datasets.""" + dummy_jsonl = tmp_path / "dummy.jsonl" + dummy_jsonl.write_text('{"prompt": "hello"}\n') + fake_acc_df = pd.DataFrame( + [{"instance_id": "repo__repo-0", "prompt": "Fix bug 0"}] + ) + config = OfflineConfig( + endpoint_config={"endpoints": ["http://test:8000"]}, + model_params={"name": "test-model"}, + datasets=[ + {"type": "performance", "path": str(dummy_jsonl)}, + { + "name": "swe_bench", + "type": "accuracy", + "accuracy_config": {"eval_method": "swe_bench_scorer"}, + }, + ], + ) + with ( + patch.object(SWEBench, "generate", return_value=fake_acc_df), + patch.object( + execute_mod, + "_resolve_accuracy_components", + return_value=(_FailingPreflightScorer, None), + ), + pytest.raises(SetupError, match="mock preflight failure"), + ): + _load_datasets(config, tmp_path, TestMode.ACC) + + @pytest.mark.unit + @pytest.mark.parametrize( + "datasets, patch_target", + [ + ( + [ + { + "type": "performance", + }, + { + "name": "swe_bench", + "type": "accuracy", + "accuracy_config": {"eval_method": "swe_bench_scorer"}, + }, + ], + "swe_bench", + ), + ( + [ + { + "type": "performance", + "accuracy_config": {"eval_method": "swe_bench_scorer"}, + }, + ], + "resolver", + ), + ], + ) + def test_perf_mode_skips_accuracy_setup(self, tmp_path, datasets, patch_target): + dummy_jsonl = tmp_path / "dummy.jsonl" + dummy_jsonl.write_text('{"text_input": "hello"}\n') + resolved_datasets = [] + for dataset in datasets: + if dataset["type"] == "performance": + resolved = { + "type": "performance", + "path": str(dummy_jsonl), + "parser": {"prompt": "text_input"}, + } + if accuracy_config := dataset.get("accuracy_config"): + resolved["accuracy_config"] = accuracy_config + resolved_datasets.append(resolved) + else: + resolved_datasets.append(dataset) + config = OfflineConfig( + endpoint_config={"endpoints": ["http://test:8000"]}, + model_params={"name": "test-model"}, + datasets=resolved_datasets, + ) + + if patch_target == "swe_bench": + with ( + patch.object(SWEBenchScorer, "preflight") as mock_preflight, + patch.object(SWEBench, "generate") as mock_generate, + ): + _, accuracy_datasets, eval_configs = _load_datasets( + config, tmp_path, TestMode.PERF + ) + mock_preflight.assert_not_called() + mock_generate.assert_not_called() + else: + with patch.object( + execute_mod, "_resolve_accuracy_components" + ) as mock_resolve: + _, accuracy_datasets, eval_configs = _load_datasets( + config, tmp_path, TestMode.PERF + ) + mock_resolve.assert_not_called() + + assert accuracy_datasets == [] + assert eval_configs == [] + + @pytest.mark.unit + @pytest.mark.parametrize("test_mode", [TestMode.ACC, TestMode.BOTH]) + def test_accuracy_modes_load_accuracy_dataset_and_preflight( + self, tmp_path, test_mode + ): + dummy_jsonl = tmp_path / "dummy.jsonl" + dummy_jsonl.write_text('{"text_input": "hello"}\n') + fake_acc_df = pd.DataFrame( + [{"instance_id": "repo__repo-0", "prompt": "Fix bug 0"}] + ) + config = OfflineConfig( + endpoint_config={"endpoints": ["http://test:8000"]}, + model_params={"name": "test-model"}, + datasets=[ + { + "type": "performance", + "path": str(dummy_jsonl), + "parser": {"prompt": "text_input"}, + }, + { + "name": "swe_bench", + "type": "accuracy", + "accuracy_config": {"eval_method": "swe_bench_scorer"}, + }, + ], + ) + + with ( + patch.object(SWEBenchScorer, "preflight") as mock_preflight, + patch.object(SWEBench, "generate", return_value=fake_acc_df), + ): + _, accuracy_datasets, eval_configs = _load_datasets( + config, tmp_path, test_mode + ) + + mock_preflight.assert_called_once_with({}, loaded_sample_count=1) + assert len(accuracy_datasets) == 1 + assert len(eval_configs) == 1 + assert eval_configs[0].scorer is SWEBenchScorer + + @pytest.mark.unit + def test_swe_bench_loader_receives_subset_and_split(self, tmp_path): + dummy_jsonl = tmp_path / "dummy.jsonl" + dummy_jsonl.write_text('{"text_input": "hello"}\n') + captured: dict[str, str] = {} + + def fake_generate( + *, + datasets_dir, + subset="verified", + split="test", + force=False, + ): + captured["subset"] = subset + captured["split"] = split + return pd.DataFrame( + [{"instance_id": "repo__repo-0", "prompt": "Fix bug 0"}] + ) + + config = OfflineConfig( + endpoint_config={"endpoints": ["http://test:8000"]}, + model_params={"name": "test-model"}, + datasets=[ + { + "type": "performance", + "path": str(dummy_jsonl), + "parser": {"prompt": "text_input"}, + }, + { + "name": "swe_bench", + "type": "accuracy", + "accuracy_config": { + "eval_method": "swe_bench_scorer", + "extras": {"subset": "lite", "split": "dev"}, + }, + }, + ], + ) + + with ( + patch.object(SWEBenchScorer, "preflight", return_value=None), + patch.object(SWEBench, "generate", side_effect=fake_generate), + ): + _load_datasets(config, tmp_path, TestMode.ACC) + + assert captured == {"subset": "lite", "split": "dev"} + + @pytest.mark.unit + def test_swe_bench_rejects_num_repeats_greater_than_one(self, tmp_path): + dummy_jsonl = tmp_path / "dummy.jsonl" + dummy_jsonl.write_text('{"text_input": "hello"}\n') + config = OfflineConfig( + endpoint_config={"endpoints": ["http://test:8000"]}, + model_params={"name": "test-model"}, + datasets=[ + { + "type": "performance", + "path": str(dummy_jsonl), + "parser": {"prompt": "text_input"}, + }, + { + "name": "swe_bench", + "type": "accuracy", + "accuracy_config": { + "eval_method": "swe_bench_scorer", + "num_repeats": 2, + }, + }, + ], + ) + + with pytest.raises( + InputValidationError, match=r"accuracy_config\.num_repeats must be 1" + ): + _load_datasets(config, tmp_path, TestMode.ACC) + + class TestYAMLTemplateValidation: """Validate all bundled YAML templates parse correctly.""" @@ -622,8 +1024,6 @@ class TestAggregatorArgs: """Tests that metrics aggregator subprocess args are correctly forwarded.""" def _make_ctx(self, config, tmp_path): - import random - rt = RuntimeSettings( metric_target=Throughput(10.0), reported_metrics=[Throughput(10.0)], @@ -1312,6 +1712,27 @@ def test_accuracy_issuer_logs_poisson_when_perf_poisson( "target_concurrency" not in m for m in load_mode_msgs ), load_mode_msgs + @pytest.mark.unit + def test_skip_endpoint_phase_omits_accuracy_phase( + self, base_rt_settings, simple_dataset + ): + config = OfflineConfig(**_OFFLINE_KWARGS) + ctx = self._make_ctx(config, base_rt_settings, simple_dataset) + ctx.eval_configs = [ + AccuracyConfiguration( + scorer=_SelfContainedScorer, + extractor=None, + dataset_name="acc", + dataset=simple_dataset, + report_dir=Path("/tmp"), + ground_truth_column=None, + num_repeats=1, + ) + ] + phases = _build_phases(ctx) + + assert all(p.phase_type != PhaseType.ACCURACY for p in phases) + @pytest.mark.unit def test_warmup_uses_independent_rng_instances( self, base_rt_settings, simple_dataset @@ -1391,13 +1812,105 @@ def make_rt(): ) +class TestFinalizeBenchmark: + def _make_eval_config(self, scorer, dataset, report_dir: Path): + return AccuracyConfiguration( + scorer=scorer, + extractor=None, + dataset_name="external_accuracy", + dataset=dataset, + report_dir=report_dir, + ground_truth_column=None, + num_repeats=1, + ) + + @pytest.mark.unit + def test_perf_mode_skips_accuracy_scoring_and_omits_scores(self, tmp_path): + config = OfflineConfig(**_OFFLINE_KWARGS) + dataset = _make_loaded_dataset() + ctx = _make_benchmark_context( + config=config, + report_dir=tmp_path, + test_mode=TestMode.PERF, + dataloader=dataset, + eval_configs=[ + AccuracyConfiguration( + scorer=_ScorerShouldNotRun, + extractor=None, + dataset_name="accuracy", + dataset=dataset, + report_dir=tmp_path, + ground_truth_column=None, + num_repeats=1, + ) + ], + ) + + finalize_benchmark(ctx, _make_benchmark_result(tmp_path)) + + assert not (tmp_path / "accuracy" / "accuracy_results.json").exists() + + @pytest.mark.unit + def test_skip_endpoint_phase_scorer_gets_empty_sample_idx_map(self, tmp_path): + config = OfflineConfig(**_OFFLINE_KWARGS) + dataset = _make_loaded_dataset() + ctx = _make_benchmark_context( + config=config, + report_dir=tmp_path, + test_mode=TestMode.ACC, + dataloader=dataset, + eval_configs=[ + self._make_eval_config(_SelfContainedScorer, dataset, tmp_path) + ], + ) + + finalize_benchmark(ctx, _make_benchmark_result(tmp_path)) + + idx_map = json.loads((tmp_path / "sample_idx_map.json").read_text()) + assert idx_map["performance"] == {"uuid-0": 0, "uuid-1": 1, "uuid-2": 2} + assert idx_map["external_accuracy"] == {} + results = json.loads( + (tmp_path / "accuracy" / "accuracy_results.json").read_text() + ) + assert results["accuracy_scores"][0]["dataset_name"] == "external_accuracy" + assert results["accuracy_scores"][0]["score"] == 1.0 + + @pytest.mark.unit + @pytest.mark.parametrize(("dataset_size", "expected"), [(3, 2), (1, 1)]) + def test_skip_endpoint_phase_scorer_reports_external_sample_count( + self, tmp_path, dataset_size, expected + ): + """External sample counts are bounded by the loaded dataset.""" + config = OfflineConfig(**_OFFLINE_KWARGS) + dataset = _make_loaded_dataset(dataset_size) + ctx = _make_benchmark_context( + config=config, + report_dir=tmp_path, + test_mode=TestMode.ACC, + dataloader=dataset, + eval_configs=[ + self._make_eval_config(_ExternalCountScorer, dataset, tmp_path) + ], + ) + + finalize_benchmark(ctx, _make_benchmark_result(tmp_path)) + + results = json.loads( + (tmp_path / "accuracy" / "accuracy_results.json").read_text() + ) + assert results["accuracy_scores"][0]["dataset_name"] == "external_accuracy" + assert results["accuracy_scores"][0]["unit_samples"] == expected + assert results["accuracy_scores"][0]["total_samples"] == expected + + class TestScorerMethodSync: """Ensure ScorerMethod enum stays in sync with the scorer registry.""" @pytest.mark.unit def test_scorer_enum_matches_registry(self): enum_values = {m.value for m in ScorerMethod} - registry_keys = set(Scorer.PREDEFINED.keys()) + # Exclude test-only scorers (ids starting with "_") + registry_keys = {k for k in Scorer.PREDEFINED if not k.startswith("_")} assert enum_values == registry_keys, ( f"ScorerMethod enum out of sync with Scorer registry.\n" f" In enum only: {enum_values - registry_keys}\n" @@ -1824,6 +2337,71 @@ def test_cancel_is_idempotent(self): assert handle.cancelled is True +class TestSetupBenchmarkExternalSampleCountLogging: + """setup_benchmark logs declared external counts for self-contained scorers.""" + + @pytest.mark.unit + def test_logs_external_sample_count_for_skip_endpoint_phase_scorer( + self, tmp_path, caplog + ): + dataset = Dataset(pd.DataFrame({"prompt": ["q0"]})) + dataset.load() + config = OfflineConfig(**_OFFLINE_KWARGS, report_dir=str(tmp_path)) + rt_settings = RuntimeSettings( + metric_target=Throughput(10.0), + reported_metrics=[Throughput(10.0)], + min_duration_ms=0, + max_duration_ms=None, + n_samples_from_dataset=1, + n_samples_to_issue=None, + min_sample_count=1, + rng_sched=random.Random(0), + rng_sample_index=random.Random(0), + load_pattern=LoadPattern(type=LoadPatternType.MAX_THROUGHPUT), + ) + eval_configs = [ + AccuracyConfiguration( + scorer=_ExternalCountScorer, + extractor=None, + dataset_name="external_accuracy", + dataset=dataset, + report_dir=tmp_path, + ground_truth_column=None, + num_repeats=1, + ) + ] + with ( + patch( + "inference_endpoint.commands.benchmark.execute.pin_loadgen", + return_value=None, + ), + patch( + "inference_endpoint.config.schema.BenchmarkConfig.to_yaml_file", + return_value=None, + ), + patch( + "inference_endpoint.commands.benchmark.execute._check_tokenizer_exists", + return_value=False, + ), + patch( + "inference_endpoint.commands.benchmark.execute._load_datasets", + return_value=(dataset, [], eval_configs), + ), + patch( + "inference_endpoint.commands.benchmark.execute.RuntimeSettings.from_config", + return_value=rt_settings, + ), + caplog.at_level("INFO"), + ): + setup_benchmark(config, TestMode.ACC) + + assert any( + "external_accuracy" in message + and "1 instances evaluated externally" in message + for message in caplog.messages + ) + + class TestProfilingHelpers: @pytest.mark.unit @pytest.mark.parametrize( @@ -2010,7 +2588,7 @@ def test_override_propagates_to_loaded_rows(self, tmp_path): config = self._build_config( perf_path, acc_path, acc_override={"max_new_tokens": 32768} ) - perf_ds, acc_datasets, _ = _load_datasets(config, tmp_path, TestMode.PERF) + perf_ds, acc_datasets, _ = _load_datasets(config, tmp_path, TestMode.BOTH) assert perf_ds.load_sample(0)[self.max_tokens_key] == 1024 assert acc_datasets[0].load_sample(0)[self.max_tokens_key] == 32768 @@ -2019,7 +2597,7 @@ def test_no_override_inherits_global(self, tmp_path): """Without overrides, both datasets use the global model_params.""" perf_path, acc_path = self._write_fixture(tmp_path) config = self._build_config(perf_path, acc_path, acc_override=None) - perf_ds, acc_datasets, _ = _load_datasets(config, tmp_path, TestMode.PERF) + perf_ds, acc_datasets, _ = _load_datasets(config, tmp_path, TestMode.BOTH) assert perf_ds.load_sample(0)[self.max_tokens_key] == 1024 assert acc_datasets[0].load_sample(0)[self.max_tokens_key] == 1024 @@ -2034,7 +2612,7 @@ def test_perf_dataset_override_also_honored(self, tmp_path): acc_override={"max_new_tokens": 32768}, perf_override={"max_new_tokens": 10240}, ) - perf_ds, acc_datasets, _ = _load_datasets(config, tmp_path, TestMode.PERF) + perf_ds, acc_datasets, _ = _load_datasets(config, tmp_path, TestMode.BOTH) assert perf_ds.load_sample(0)[self.max_tokens_key] == 10240 assert acc_datasets[0].load_sample(0)[self.max_tokens_key] == 32768 diff --git a/tests/unit/commands/test_score_accuracy.py b/tests/unit/commands/test_score_accuracy.py index 1ca7a0be7..835b73c2b 100644 --- a/tests/unit/commands/test_score_accuracy.py +++ b/tests/unit/commands/test_score_accuracy.py @@ -15,6 +15,8 @@ """Tests for per-dataset accuracy scoring in finalize_benchmark.""" +# ruff: noqa: I001 + from __future__ import annotations import sys @@ -29,6 +31,7 @@ _phase_response_counts, _score_accuracy, ) +from inference_endpoint.config import schema as config_schema from inference_endpoint.config.schema import DatasetType # Module object for the tests that monkeypatch execute's own module-level symbols @@ -51,6 +54,8 @@ def num_samples(self) -> int: class _FakeScorer: """Duck-typed scorer stand-in with no breakdown.""" + SKIP_ENDPOINT_PHASE = False + def __init__( self, name, dataset, report_dir, extractor=None, ground_truth_column=None, **x ): @@ -158,11 +163,19 @@ def _by_name(scores: list[dict]) -> dict[str, dict]: return {e["dataset_name"]: e for e in scores} -def _ctx(cfgs, tokenizer_name=None, report_dir=None): +def _ctx( + cfgs, + tokenizer_name=None, + report_dir=None, + test_mode: config_schema.TestMode = config_schema.TestMode.ACC, +): # tokenizer_name None => OSL is skipped (fake scorers have no get_raw_outputs). # report_dir None => the uuid bound falls back to an unbounded read (no map). return SimpleNamespace( - eval_configs=cfgs, tokenizer_name=tokenizer_name, report_dir=report_dir + eval_configs=cfgs, + tokenizer_name=tokenizer_name, + report_dir=report_dir, + test_mode=test_mode, ) @@ -235,6 +248,13 @@ def test_performance_entry_uses_issued_count_for_total(self, tmp_path): def test_empty_when_no_datasets(self, tmp_path): assert _score_accuracy(_ctx([]), _RESULT) == [] + def test_perf_mode_skips_accuracy_scoring(self, tmp_path): + cfg = _cfg("aime25::gptoss", 30, 0.8, tmp_path) + assert ( + _score_accuracy(_ctx([cfg], test_mode=config_schema.TestMode.PERF), _RESULT) + == [] + ) + def test_no_osl_without_tokenizer(self, tmp_path): # tokenizer_name None (the default) => no output_sequence_lengths attached, # and the OSL path is never entered (fake scorers have no get_raw_outputs). diff --git a/tests/unit/dataset_manager/test_swe_bench_dataset.py b/tests/unit/dataset_manager/test_swe_bench_dataset.py new file mode 100644 index 000000000..11f55724e --- /dev/null +++ b/tests/unit/dataset_manager/test_swe_bench_dataset.py @@ -0,0 +1,159 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Unit tests for SWEBench predefined dataset.""" + +from pathlib import Path +from unittest.mock import patch + +import pandas as pd +import pytest +from inference_endpoint.dataset_manager.dataset import Dataset +from inference_endpoint.dataset_manager.predefined.swe_bench import SWEBench + +pytestmark = pytest.mark.unit + +_FAKE_INSTANCES = [ + {"instance_id": f"repo__repo-{i}", "problem_statement": f"Fix bug {i}"} + for i in range(5) +] + + +def _make_hf_df() -> pd.DataFrame: + return pd.DataFrame(_FAKE_INSTANCES) + + +class TestSWEBenchRegistration: + def test_registered(self): + assert "swe_bench" in Dataset.PREDEFINED + assert Dataset.PREDEFINED["swe_bench"] is SWEBench + + def test_accuracy_only_flag(self): + assert SWEBench.ACCURACY_ONLY is True + + @pytest.mark.parametrize( + ("subset", "expected"), + [ + ("verified", "princeton-nlp/SWE-bench_Verified"), + ("lite", "princeton-nlp/SWE-bench_Lite"), + ], + ) + def test_hf_dataset_name(self, subset: str, expected: str): + assert SWEBench.hf_dataset_name(subset) == expected + + def test_hf_dataset_name_invalid_subset_raises(self): + with pytest.raises(ValueError, match="Unknown SWE-bench subset"): + SWEBench.hf_dataset_name("invalid") + + +class TestSWEBenchGenerate: + def test_downloads_and_caches(self, tmp_path: Path): + with patch( + "inference_endpoint.dataset_manager.predefined.swe_bench.load_from_huggingface", + return_value=_make_hf_df(), + ) as mock_hf: + df1 = SWEBench.generate(datasets_dir=tmp_path) + + assert mock_hf.call_count == 1 + assert list(df1.columns) == ["instance_id", "prompt"] + assert len(df1) == 5 + assert df1["prompt"].iloc[0] == "Fix bug 0" + + # Second call should hit parquet cache, not HF + with patch( + "inference_endpoint.dataset_manager.predefined.swe_bench.load_from_huggingface", + ) as mock_hf2: + df2 = SWEBench.generate(datasets_dir=tmp_path) + + mock_hf2.assert_not_called() + assert list(df2.columns) == ["instance_id", "prompt"] + assert len(df2) == 5 + + def test_unknown_subset_raises(self, tmp_path: Path): + with pytest.raises(ValueError, match="Unknown SWE-bench subset"): + SWEBench.generate(datasets_dir=tmp_path, subset="invalid") + + def test_force_regenerate(self, tmp_path: Path): + with patch( + "inference_endpoint.dataset_manager.predefined.swe_bench.load_from_huggingface", + return_value=_make_hf_df(), + ) as mock_hf: + SWEBench.generate(datasets_dir=tmp_path) + assert mock_hf.call_count == 1 + + SWEBench.generate(datasets_dir=tmp_path, force=True) + assert mock_hf.call_count == 2 + + @pytest.mark.parametrize( + ("subset", "split", "expected_repo"), + [ + ("lite", "dev", "princeton-nlp/SWE-bench_Lite"), + ("verified", "test", "princeton-nlp/SWE-bench_Verified"), + ], + ) + def test_subset_and_split_select_hf_repo_and_cache_directory( + self, tmp_path: Path, subset: str, split: str, expected_repo: str + ): + with patch( + "inference_endpoint.dataset_manager.predefined.swe_bench.load_from_huggingface", + return_value=_make_hf_df(), + ) as mock_hf: + df = SWEBench.generate(datasets_dir=tmp_path, subset=subset, split=split) + + assert mock_hf.call_args.args == (expected_repo,) + assert mock_hf.call_args.kwargs["split"] == split + assert mock_hf.call_args.kwargs["cache_dir"] == ( + tmp_path / "hf_cache" / f"swe_bench_{subset}_{split}" + ) + assert len(df) == 5 + cache_dir = tmp_path / "swe_bench" / subset / split + assert len(list(cache_dir.glob("*.parquet"))) == 1 + + def test_corrupt_parquet_cache_reports_cache_path(self, tmp_path: Path): + cache_path = ( + tmp_path + / "swe_bench" + / "verified" + / "test" + / "swe_bench_verified_test.parquet" + ) + cache_path.parent.mkdir(parents=True) + cache_path.write_text("not parquet") + + with ( + patch( + "inference_endpoint.dataset_manager.predefined.swe_bench.pd.read_parquet", + side_effect=ValueError("bad parquet"), + ), + pytest.raises(RuntimeError, match="appears corrupt"), + ): + SWEBench.generate(datasets_dir=tmp_path) + + def test_huggingface_load_failure_is_logged_and_reraised( + self, tmp_path: Path, caplog + ): + error = RuntimeError("hf unavailable") + with ( + patch( + "inference_endpoint.dataset_manager.predefined.swe_bench.load_from_huggingface", + side_effect=error, + ), + pytest.raises(RuntimeError, match="hf unavailable") as exc_info, + caplog.at_level("ERROR"), + ): + SWEBench.generate(datasets_dir=tmp_path, subset="lite", split="dev") + + assert exc_info.value is error + assert "Error loading SWE-bench lite/dev from HuggingFace" in caplog.text diff --git a/tests/unit/evaluation/swebench_service/test_runner.py b/tests/unit/evaluation/swebench_service/test_runner.py new file mode 100644 index 000000000..6bc1e9b02 --- /dev/null +++ b/tests/unit/evaluation/swebench_service/test_runner.py @@ -0,0 +1,257 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +import os +import sys +import threading +from pathlib import Path + +import msgspec.json +import pytest +import yaml + +_SERVICE_ROOT = ( + Path(__file__).resolve().parents[4] + / "src" + / "inference_endpoint" + / "evaluation" + / "swebench_service" +) +sys.path.insert(0, str(_SERVICE_ROOT)) + +from swebench_service import runner as runner_mod # noqa: E402 +from swebench_service.runner import ( # noqa: E402 + CancellationToken, + RunCancelled, + RunnerError, + SwebenchRunner, +) +from swebench_service.schemas import RunRequest # noqa: E402 + +pytestmark = pytest.mark.unit + + +def _request(endpoints: list[str]) -> RunRequest: + return RunRequest( + model_name="test-model", + endpoint_urls=endpoints, + endpoint_api_key=None, + generation_params={"name": "test-model"}, + subset="lite", + split="test", + num_instances=1, + workers=1, + max_eval_workers=1, + evaluated_instance_ids=["repo__repo-1"], + ) + + +def test_run_subprocess_streams_output_to_log(tmp_path): + log_path = tmp_path / "subprocess.log" + + runner_mod._run_subprocess( + [sys.executable, "-c", "print('first'); print('second')"], + log_path, + cwd=tmp_path, + timeout_s=5, + ) + + assert log_path.read_text() == "first\nsecond\n" + + +def test_run_subprocess_reports_bounded_failure_tail(tmp_path): + log_path = tmp_path / "subprocess.log" + script = ( + "import sys\n" + "print('early-marker')\n" + "for i in range(700): print(f'{i:04d}-' + 'x' * 100)\n" + "print('final-marker')\n" + "sys.exit(7)\n" + ) + + with pytest.raises(RunnerError, match="exited with code 7") as exc_info: + runner_mod._run_subprocess( + [sys.executable, "-c", script], + log_path, + cwd=tmp_path, + timeout_s=5, + ) + + assert "early-marker" in log_path.read_text() + failure_tail = str(exc_info.value).partition("\n")[2] + assert "final-marker" in failure_tail + assert "early-marker" not in failure_tail + + +def test_run_subprocess_timeout_preserves_partial_log(tmp_path): + log_path = tmp_path / "subprocess.log" + + with pytest.raises(RunnerError, match="timed out after 1s"): + runner_mod._run_subprocess( + [ + sys.executable, + "-c", + "import time; print('started', flush=True); time.sleep(30)", + ], + log_path, + cwd=tmp_path, + timeout_s=1, + ) + + assert log_path.read_text() == "started\n" + + +def test_run_subprocess_cancellation_preserves_partial_log(tmp_path): + log_path = tmp_path / "subprocess.log" + cancel_token = CancellationToken() + cancel_timer = threading.Timer(0.2, cancel_token.cancel) + cancel_timer.start() + try: + with pytest.raises(RunCancelled, match="subprocess cancelled"): + runner_mod._run_subprocess( + [ + sys.executable, + "-c", + "import time; print('started', flush=True); time.sleep(30)", + ], + log_path, + cwd=tmp_path, + timeout_s=5, + cancel_token=cancel_token, + ) + finally: + cancel_timer.cancel() + cancel_timer.join() + + assert log_path.read_text() == "started\n" + + +def test_base_env_keeps_proxies_and_sets_no_proxy_for_loopback(monkeypatch, tmp_path): + monkeypatch.setenv("http_proxy", "http://proxy.example:8080") + monkeypatch.setenv("https_proxy", "http://proxy.example:8080") + monkeypatch.setenv("HTTP_PROXY", "http://proxy.example:8080") + monkeypatch.setenv("HTTPS_PROXY", "http://proxy.example:8080") + monkeypatch.setenv("all_proxy", "socks5://proxy.example:1080") + monkeypatch.setenv("ALL_PROXY", "socks5://proxy.example:1080") + monkeypatch.setenv("NO_PROXY", "intel.com") + + runner = SwebenchRunner(project_root=tmp_path, subprocess_timeout_s=30) + env = runner._base_env(_request(["http://localhost:30000"])) + + assert env["http_proxy"] == "http://proxy.example:8080" + assert env["https_proxy"] == "http://proxy.example:8080" + assert env["HTTP_PROXY"] == "http://proxy.example:8080" + assert env["HTTPS_PROXY"] == "http://proxy.example:8080" + assert env["all_proxy"] == "socks5://proxy.example:1080" + assert env["ALL_PROXY"] == "socks5://proxy.example:1080" + assert {"127.0.0.1", "localhost", "intel.com"} <= set(env["NO_PROXY"].split(",")) + assert env["NO_PROXY"] == env["no_proxy"] + + +def test_base_env_keeps_proxies_for_non_loopback_endpoints(monkeypatch, tmp_path): + monkeypatch.setenv("https_proxy", "http://proxy.example:8080") + + runner = SwebenchRunner(project_root=tmp_path, subprocess_timeout_s=30) + env = runner._base_env(_request(["http://swebench-host:30000"])) + + assert env["https_proxy"] == "http://proxy.example:8080" + assert "swebench-host" in env["NO_PROXY"].split(",") + + +def test_patch_config_rewrites_localhost_api_base_to_127_0_0_1(tmp_path): + runner = SwebenchRunner(project_root=tmp_path, subprocess_timeout_s=30) + + patched = runner._patch_config( + tmp_path, + _request(["http://localhost:30000"]), + ) + + cfg = yaml.safe_load(patched.read_text()) + assert cfg["model"]["model_kwargs"]["api_base"] == "http://127.0.0.1:30000/v1" + assert cfg["model"]["model_kwargs"]["api_key"] == "EMPTY" + + +def test_run_agent_filters_exact_instance_ids(monkeypatch, tmp_path): + commands: list[list[str]] = [] + + def fake_run_subprocess(cmd, *args, **kwargs): + commands.append(cmd) + + monkeypatch.setattr(runner_mod, "_run_subprocess", fake_run_subprocess) + request = _request(["http://endpoint:30000"]) + request.evaluated_instance_ids = ["repo__repo-1", "repo.with.regex+chars"] + runner = SwebenchRunner(project_root=tmp_path, subprocess_timeout_s=30) + + runner._run_agent(request, tmp_path / "config.yaml", tmp_path, tmp_path) + + cmd = commands[0] + assert "--slice" not in cmd + assert cmd[cmd.index("--filter") + 1] == ( + "^(?:repo__repo\\-1|repo\\.with\\.regex\\+chars)$" + ) + + +def test_run_agent_toolcall_patch_prepends_overlay_pythonpath(monkeypatch, tmp_path): + envs: list[dict[str, str]] = [] + overlay = tmp_path / "overlay" + request = _request(["http://endpoint:30000"]) + request.enable_swebench_toolcall_patch = True + runner = SwebenchRunner(project_root=tmp_path, subprocess_timeout_s=30) + + def fake_create_overlay(self, overlay_root, replacement_root): + assert replacement_root == self._template_dir + overlay.mkdir() + return overlay + + def fake_run_subprocess(cmd, log_path, *, env, **kwargs): + envs.append(env) + + monkeypatch.setenv("PYTHONPATH", "/existing/path") + monkeypatch.setattr( + SwebenchRunner, "_create_toolcall_patch_overlay", fake_create_overlay + ) + monkeypatch.setattr(runner_mod, "_run_subprocess", fake_run_subprocess) + + runner._run_agent(request, tmp_path / "config.yaml", tmp_path, tmp_path) + + pythonpath = envs[0]["PYTHONPATH"].split(os.pathsep) + assert pythonpath[:2] == [str(overlay), "/existing/path"] + + +def test_validate_prediction_ids_rejects_unexpected_instances(tmp_path): + request = _request(["http://endpoint:30000"]) + request.evaluated_instance_ids = ["repo__repo-1"] + preds = tmp_path / "preds.json" + preds.write_bytes( + msgspec.json.encode({"repo__repo-1": "patch", "repo__repo-2": "patch"}) + ) + runner = SwebenchRunner(project_root=tmp_path, subprocess_timeout_s=30) + + with pytest.raises(RunnerError, match="unexpected SWE-bench"): + runner._validate_prediction_ids(request, preds) + + +def test_run_eval_persists_harness_run_id(monkeypatch, tmp_path): + runner = SwebenchRunner(project_root=tmp_path, subprocess_timeout_s=30) + request = _request(["http://endpoint:30000"]) + output_dir = tmp_path / "output" + run_dir = tmp_path / "run" + output_dir.mkdir() + run_dir.mkdir() + preds_path = output_dir / "preds.json" + preds_path.write_text('{"repo__repo-1":"patch"}') + + def fake_run_subprocess(cmd, log_path, *, cwd, **kwargs): + assert cmd[:3] == [sys.executable, "-m", "swebench.harness.run_evaluation"] + run_id = cmd[cmd.index("--run_id") + 1] + assert (run_dir / "swe_bench_eval_run_id.txt").read_text() == run_id + (cwd / f"test-model.{run_id}.json").write_text( + '{"resolved_instances":1,"submitted_instances":1}' + ) + + monkeypatch.setattr(runner_mod, "_run_subprocess", fake_run_subprocess) + + result_path = runner._run_eval(request, preds_path, output_dir, run_dir) + + assert result_path.exists() + assert (run_dir / "swe_bench_eval_run_id.txt").read_text().startswith("endpoints_") diff --git a/tests/unit/evaluation/swebench_service/test_server.py b/tests/unit/evaluation/swebench_service/test_server.py new file mode 100644 index 000000000..dbc96db0f --- /dev/null +++ b/tests/unit/evaluation/swebench_service/test_server.py @@ -0,0 +1,464 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +import asyncio +import sys +import threading +import time +from pathlib import Path + +import pytest +from aiohttp import web +from aiohttp.test_utils import TestClient, TestServer + +_SERVICE_ROOT = ( + Path(__file__).resolve().parents[4] + / "src" + / "inference_endpoint" + / "evaluation" + / "swebench_service" +) +sys.path.insert(0, str(_SERVICE_ROOT)) + +from swebench_service.config import ServiceConfig # noqa: E402 +from swebench_service.schemas import RunRequest, RunStatus # noqa: E402 +from swebench_service.server import RunManager, create_app # noqa: E402 + +pytestmark = pytest.mark.unit + + +class FakeRunner: + def __init__(self, *, delay: float = 0.0, fail: bool = False): + self.delay = delay + self.fail = fail + self.requests: list[object] = [] + + def run(self, request, run_dir: Path): + self.requests.append(request) + if self.delay: + import time + + time.sleep(self.delay) + if self.fail: + raise RuntimeError("runner failed") + (run_dir / "preds.json").write_text("{}") + (run_dir / "swe_bench_results.json").write_text( + '{"resolved_instances":1,"submitted_instances":1}' + ) + return {"resolved_instances": 1, "submitted_instances": 1} + + +class AgentProgressRunner: + def __init__(self): + self.ready = threading.Event() + + def run(self, request, run_dir: Path): + output_dir = run_dir / "swe_bench_output" + output_dir.mkdir() + (output_dir / "exit_statuses_0002.yaml").write_text( + "\n".join( + [ + "instances_by_exit_status:", + " submitted:", + " - repo__repo-1", + " resolved:", + " - repo__repo-2", + "", + ] + ) + ) + self.ready.set() + time.sleep(0.2) + (run_dir / "preds.json").write_text("{}") + (run_dir / "swe_bench_results.json").write_text( + '{"resolved_instances":1,"submitted_instances":2}' + ) + return {"resolved_instances": 1, "submitted_instances": 2} + + +class EvalProgressRunner: + def __init__(self): + self.ready = threading.Event() + + def run(self, request, run_dir: Path): + output_dir = run_dir / "swe_bench_output" + report_dir = ( + output_dir / "logs" / "run_evaluation" / "eval-run-1" / "test-model" + ) + report_dir.mkdir(parents=True) + (run_dir / "swe_bench_eval_run_id.txt").write_text("eval-run-1") + (output_dir / "preds.json").write_text( + '{"repo__repo-1":"patch","repo__repo-2":"patch","repo__repo-3":"patch"}' + ) + (report_dir / "repo__repo-1" / "report.json").parent.mkdir() + (report_dir / "repo__repo-1" / "report.json").write_text("{}") + (report_dir / "repo__repo-2" / "report.json").parent.mkdir() + (report_dir / "repo__repo-2" / "report.json").write_text("{}") + self.ready.set() + time.sleep(0.2) + (run_dir / "preds.json").write_text("{}") + (run_dir / "swe_bench_results.json").write_text( + '{"resolved_instances":2,"submitted_instances":3}' + ) + return {"resolved_instances": 2, "submitted_instances": 3} + + +class CancellationAwareRunner: + def __init__(self): + self.started = threading.Event() + self.cancelled = threading.Event() + + def run(self, request, run_dir: Path, cancel_token=None): + self.started.set() + deadline = time.monotonic() + 5 + while ( + cancel_token is not None + and not cancel_token.is_cancelled() + and time.monotonic() < deadline + ): + time.sleep(0.01) + if cancel_token is not None and cancel_token.is_cancelled(): + self.cancelled.set() + return {"resolved_instances": 0, "submitted_instances": 0} + + +def _payload() -> dict: + return { + "model_name": "test-model", + "endpoint_urls": ["http://endpoint"], + "endpoint_api_key": "secret", + "generation_params": {"temperature": 0.0}, + "subset": "lite", + "split": "test", + "num_instances": 1, + "workers": 1, + "max_eval_workers": 1, + "evaluated_instance_ids": ["repo__repo-1"], + } + + +def _payload_with_instances(n: int) -> dict: + payload = _payload() + payload["num_instances"] = n + payload["evaluated_instance_ids"] = [f"repo__repo-{i + 1}" for i in range(n)] + return payload + + +async def _client(tmp_path: Path, runner) -> TestClient: + app = create_app( + ServiceConfig(artifact_root=tmp_path, max_concurrent_runs=1), + runner=runner, + ) + client = TestClient(TestServer(app)) + await client.start_server() + return client + + +async def _auth_client(tmp_path: Path, runner) -> TestClient: + app = create_app( + ServiceConfig(artifact_root=tmp_path, max_concurrent_runs=1, auth_token="tok"), + runner=runner, + ) + client = TestClient(TestServer(app)) + await client.start_server() + return client + + +@pytest.mark.asyncio +async def test_health_response_schema(tmp_path): + client = await _client(tmp_path, FakeRunner()) + try: + resp = await client.get("/health") + body = await resp.json() + finally: + await client.close() + + assert resp.status == 200 + assert body["api_version"] == "v1" + assert "swebench.run" in body["capabilities"] + assert "swebench.cancel" in body["capabilities"] + assert "artifacts.download" in body["capabilities"] + assert "swebench.progress" in body["capabilities"] + + +@pytest.mark.asyncio +async def test_post_run_validates_requests(tmp_path): + client = await _client(tmp_path, FakeRunner()) + try: + resp = await client.post("/v1/runs", json={"model_name": ""}) + finally: + await client.close() + + assert resp.status == 400 + + +@pytest.mark.asyncio +async def test_post_run_rejects_multiple_endpoint_urls(tmp_path): + runner = FakeRunner() + client = await _client(tmp_path, runner) + payload = _payload() + payload["endpoint_urls"] = ["http://endpoint-a", "http://endpoint-b"] + try: + resp = await client.post("/v1/runs", json=payload) + finally: + await client.close() + + assert resp.status == 400 + assert runner.requests == [] + + +@pytest.mark.asyncio +async def test_optional_auth_token_is_enforced(tmp_path): + client = await _auth_client(tmp_path, FakeRunner()) + try: + unauthorized = await client.get("/health") + authorized = await client.get( + "/health", headers={"Authorization": "Bearer tok"} + ) + finally: + await client.close() + + assert unauthorized.status == 401 + assert authorized.status == 200 + + +@pytest.mark.asyncio +async def test_runner_transitions_to_succeeded(tmp_path): + runner = FakeRunner() + client = await _client(tmp_path, runner) + try: + submit = await client.post("/v1/runs", json=_payload()) + submitted = await submit.json() + for _ in range(20): + status_resp = await client.get(f"/v1/runs/{submitted['run_id']}") + status = await status_resp.json() + if status["status"] == "succeeded": + break + await asyncio.sleep(0.01) + finally: + await client.close() + + assert submit.status == 202 + assert status["status"] == "succeeded" + assert status["result"] == {"resolved_instances": 1, "submitted_instances": 1} + assert runner.requests[0].evaluated_instance_ids == ["repo__repo-1"] + assert status["phase"] == "succeeded" + assert status["agent_completed"] == 1 + assert status["eval_completed"] == 1 + + +@pytest.mark.asyncio +async def test_status_reports_agent_progress_from_exit_statuses(tmp_path): + runner = AgentProgressRunner() + client = await _client(tmp_path, runner) + try: + submit = await client.post("/v1/runs", json=_payload_with_instances(3)) + submitted = await submit.json() + assert await asyncio.to_thread(runner.ready.wait, 2) + + status_resp = await client.get(f"/v1/runs/{submitted['run_id']}") + status = await status_resp.json() + finally: + await client.close() + + assert status_resp.status == 200 + assert status["phase"] == "agent" + assert status["agent_total"] == 3 + assert status["agent_completed"] == 2 + assert status["eval_total"] == 0 + assert status["eval_completed"] == 0 + + +@pytest.mark.asyncio +async def test_status_reports_eval_progress_from_harness_reports(tmp_path): + runner = EvalProgressRunner() + client = await _client(tmp_path, runner) + try: + submit = await client.post("/v1/runs", json=_payload_with_instances(3)) + submitted = await submit.json() + assert await asyncio.to_thread(runner.ready.wait, 2) + + status_resp = await client.get(f"/v1/runs/{submitted['run_id']}") + status = await status_resp.json() + finally: + await client.close() + + assert status_resp.status == 200 + assert status["phase"] == "eval" + assert status["eval_total"] == 3 + assert status["eval_completed"] == 2 + + +@pytest.mark.asyncio +async def test_status_reports_zero_progress_when_files_are_missing(tmp_path): + client = await _client(tmp_path, FakeRunner(delay=0.2)) + try: + submit = await client.post("/v1/runs", json=_payload_with_instances(2)) + submitted = await submit.json() + + status_resp = await client.get(f"/v1/runs/{submitted['run_id']}") + status = await status_resp.json() + finally: + await client.close() + + assert status_resp.status == 200 + assert status["phase"] in {"queued", "agent"} + assert status["agent_total"] == 2 + assert status["agent_completed"] == 0 + assert status["eval_completed"] == 0 + + +@pytest.mark.asyncio +async def test_runner_transitions_to_failed(tmp_path): + client = await _client(tmp_path, FakeRunner(fail=True)) + try: + submit = await client.post("/v1/runs", json=_payload()) + submitted = await submit.json() + for _ in range(20): + status_resp = await client.get(f"/v1/runs/{submitted['run_id']}") + status = await status_resp.json() + if status["status"] == "failed": + break + await asyncio.sleep(0.01) + finally: + await client.close() + + assert status["status"] == "failed" + assert "runner failed" in status["error"] + + +@pytest.mark.asyncio +async def test_cancel_run_marks_cancelled_and_signals_runner(tmp_path): + runner = CancellationAwareRunner() + client = await _client(tmp_path, runner) + try: + submit = await client.post("/v1/runs", json=_payload()) + submitted = await submit.json() + assert await asyncio.to_thread(runner.started.wait, 2) + + cancel_resp = await client.post(f"/v1/runs/{submitted['run_id']}/cancel") + cancelled = await cancel_resp.json() + + assert await asyncio.to_thread(runner.cancelled.wait, 2) + finally: + await client.close() + + assert cancel_resp.status == 200 + assert cancelled["status"] == "cancelled" + + +@pytest.mark.asyncio +async def test_shutdown_cancels_active_runs(tmp_path): + runner = CancellationAwareRunner() + client = await _client(tmp_path, runner) + await client.post("/v1/runs", json=_payload()) + assert await asyncio.to_thread(runner.started.wait, 2) + + await client.close() + + assert await asyncio.to_thread(runner.cancelled.wait, 2) + + +@pytest.mark.asyncio +async def test_bounded_concurrency_returns_429(tmp_path): + client = await _client(tmp_path, FakeRunner(delay=0.2)) + try: + first = await client.post("/v1/runs", json=_payload()) + second = await client.post("/v1/runs", json=_payload()) + finally: + await client.close() + + assert first.status == 202 + assert second.status == 429 + + +@pytest.mark.asyncio +async def test_concurrent_submit_reserves_capacity_once(tmp_path): + manager = RunManager( + config=ServiceConfig(artifact_root=tmp_path, max_concurrent_runs=1), + runner=FakeRunner(delay=0.2), + ) + try: + manager_request = RunRequest.model_validate(_payload()) + results = await asyncio.gather( + manager.submit(manager_request), + manager.submit(manager_request), + return_exceptions=True, + ) + finally: + await manager.cancel_all_active() + + accepted = [result for result in results if isinstance(result, RunStatus)] + rejected = [ + result for result in results if isinstance(result, web.HTTPTooManyRequests) + ] + assert len(accepted) == 1 + assert len(rejected) == 1 + assert manager.active_count() <= 1 + + +@pytest.mark.asyncio +async def test_artifact_endpoint_blocks_path_traversal(tmp_path): + client = await _client(tmp_path, FakeRunner()) + try: + submit = await client.post("/v1/runs", json=_payload()) + submitted = await submit.json() + run_id = submitted["run_id"] + for _ in range(20): + status_resp = await client.get(f"/v1/runs/{run_id}") + status = await status_resp.json() + if status["status"] == "succeeded": + break + await asyncio.sleep(0.01) + ok = await client.get(f"/v1/runs/{run_id}/artifacts/preds.json") + blocked = await client.get(f"/v1/runs/{run_id}/artifacts/..%2Fstatus.json") + finally: + await client.close() + + assert ok.status == 200 + assert blocked.status == 404 + + +@pytest.mark.asyncio +async def test_status_redacts_api_keys(tmp_path): + client = await _client(tmp_path, FakeRunner(delay=0.05)) + try: + submit = await client.post("/v1/runs", json=_payload()) + submitted = await submit.json() + status_path = tmp_path / submitted["run_id"] / "status.json" + request_path = tmp_path / submitted["run_id"] / "request.json" + for _ in range(100): + if request_path.exists(): + break + await asyncio.sleep(0.01) + request_text = request_path.read_text() + status_text = status_path.read_text() + finally: + await client.close() + + assert "secret" not in request_text + assert "" in request_text + assert "secret" not in status_text + + +@pytest.mark.asyncio +async def test_failed_status_redacts_secret_values(tmp_path): + class SecretFailRunner: + def run(self, request, run_dir: Path): + raise RuntimeError(f"failed with api_key={request.endpoint_api_key}") + + client = await _client(tmp_path, SecretFailRunner()) + try: + submit = await client.post("/v1/runs", json=_payload()) + submitted = await submit.json() + for _ in range(20): + status_resp = await client.get(f"/v1/runs/{submitted['run_id']}") + status = await status_resp.json() + if status["status"] == "failed": + break + await asyncio.sleep(0.01) + finally: + await client.close() + + assert "secret" not in status["error"] + assert "" in status["error"] diff --git a/tests/unit/evaluation/test_actions_toolcall.py b/tests/unit/evaluation/test_actions_toolcall.py new file mode 100644 index 000000000..19f063d09 --- /dev/null +++ b/tests/unit/evaluation/test_actions_toolcall.py @@ -0,0 +1,213 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Unit tests for the Qwen mini-swe-agent toolcall replacements.""" + +import importlib.util +import json +import os +import sys +import types +from pathlib import Path +from types import SimpleNamespace + +import pytest + +pytestmark = pytest.mark.unit + +_TEMPLATES_DIR = ( + Path(__file__).resolve().parents[3] + / "src" + / "inference_endpoint" + / "evaluation" + / "swebench_service" + / "swebench_service" + / "templates" +) +_ACTIONS_TOOLCALL = _TEMPLATES_DIR / "actions_toolcall.py" +_LITELLM_MODEL = _TEMPLATES_DIR / "litellm_model.py" + + +def _load_actions_module(monkeypatch): + class FormatError(Exception): + pass + + exceptions_mod = types.ModuleType("minisweagent.exceptions") + exceptions_mod.FormatError = FormatError + multimodal_mod = types.ModuleType("minisweagent.models.utils.openai_multimodal") + multimodal_mod.expand_multimodal_content = lambda msg, pattern: msg + + for name in [ + "minisweagent", + "minisweagent.models", + "minisweagent.models.utils", + ]: + monkeypatch.setitem(sys.modules, name, types.ModuleType(name)) + monkeypatch.setitem(sys.modules, "minisweagent.exceptions", exceptions_mod) + monkeypatch.setitem( + sys.modules, + "minisweagent.models.utils.openai_multimodal", + multimodal_mod, + ) + + spec = importlib.util.spec_from_file_location( + "_test_actions_toolcall", _ACTIONS_TOOLCALL + ) + assert spec is not None + assert spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def _load_litellm_model_module(monkeypatch): + class LitellmError(Exception): + pass + + litellm_mod = types.ModuleType("litellm") + litellm_mod.exceptions = SimpleNamespace( + UnsupportedParamsError=LitellmError, + NotFoundError=LitellmError, + PermissionDeniedError=LitellmError, + ContextWindowExceededError=LitellmError, + AuthenticationError=LitellmError, + ) + litellm_mod.utils = SimpleNamespace(register_model=lambda model: None) + + models_mod = types.ModuleType("minisweagent.models") + models_mod.GLOBAL_MODEL_STATS = SimpleNamespace(add=lambda cost: None) + actions_mod = types.ModuleType("minisweagent.models.utils.actions_toolcall") + actions_mod.TOOL_SCHEMAS = [] + actions_mod.format_toolcall_observation_messages = lambda **kwargs: [] + actions_mod.parse_toolcall_actions = lambda *args, **kwargs: [] + anthropic_mod = types.ModuleType("minisweagent.models.utils.anthropic_utils") + anthropic_mod._reorder_anthropic_thinking_blocks = lambda messages: messages + cache_mod = types.ModuleType("minisweagent.models.utils.cache_control") + cache_mod.set_cache_control = lambda messages, mode: messages + multimodal_mod = types.ModuleType("minisweagent.models.utils.openai_multimodal") + multimodal_mod.expand_multimodal_content = lambda message, pattern: message + retry_mod = types.ModuleType("minisweagent.models.utils.retry") + retry_mod.retry = lambda **kwargs: [] + + for name, module in { + "litellm": litellm_mod, + "minisweagent": types.ModuleType("minisweagent"), + "minisweagent.models": models_mod, + "minisweagent.models.utils": types.ModuleType("minisweagent.models.utils"), + "minisweagent.models.utils.actions_toolcall": actions_mod, + "minisweagent.models.utils.anthropic_utils": anthropic_mod, + "minisweagent.models.utils.cache_control": cache_mod, + "minisweagent.models.utils.openai_multimodal": multimodal_mod, + "minisweagent.models.utils.retry": retry_mod, + }.items(): + monkeypatch.setitem(sys.modules, name, module) + + spec = importlib.util.spec_from_file_location("_test_litellm_model", _LITELLM_MODEL) + assert spec is not None + assert spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def _tool_call(name: str, args: dict) -> SimpleNamespace: + return SimpleNamespace( + id="call-1", + function=SimpleNamespace(name=name, arguments=json.dumps(args)), + ) + + +def test_litellm_model_loopback_preserves_proxy_environment(monkeypatch): + proxy_vars = { + "http_proxy": "http://proxy.example:8080", + "https_proxy": "http://proxy.example:8080", + "HTTP_PROXY": "http://proxy.example:8080", + "HTTPS_PROXY": "http://proxy.example:8080", + "all_proxy": "socks5://proxy.example:1080", + "ALL_PROXY": "socks5://proxy.example:1080", + } + for name, value in proxy_vars.items(): + monkeypatch.setenv(name, value) + + litellm_model = _load_litellm_model_module(monkeypatch) + config = SimpleNamespace( + model_kwargs={"api_base": "http://127.0.0.1:30000/v1"}, + litellm_model_registry=None, + ) + + litellm_model.LitellmModel(config_class=lambda **kwargs: config) + + assert {name: os.environ[name] for name in proxy_vars} == proxy_vars + + +def test_finish_emits_relative_pathspecs_and_git_add_intent(monkeypatch): + actions_mod = _load_actions_module(monkeypatch) + + actions = actions_mod.parse_toolcall_actions( + [ + _tool_call( + "finish", + { + "files_modified": [ + "/testbed/pkg/new file.py", + "tests/test_widget.py", + ] + }, + ) + ], + format_error_template="{{ error }}", + ) + + command = actions[0]["command"] + assert ( + command == "echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT && " + "git -C /testbed add -N -- 'pkg/new file.py' tests/test_widget.py && " + "git -C /testbed diff HEAD -- 'pkg/new file.py' tests/test_widget.py" + ) + assert "/testbed/pkg/new file.py" not in command + + +def test_str_replace_editor_view_range_emits_clean_awk(monkeypatch): + actions_mod = _load_actions_module(monkeypatch) + + actions = actions_mod.parse_toolcall_actions( + [ + _tool_call( + "str_replace_editor", + { + "command": "view", + "path": "/testbed/pkg/file.py", + "view_range": [2, 4], + }, + ) + ], + format_error_template="{{ error }}", + ) + + command = actions[0]["command"] + assert ( + command == "awk 'NR>=2 && NR<=4 " + '{printf "%6d\\t%s\\n", NR, $0}\' /testbed/pkg/file.py' + ) + assert r"\'" not in command + + +def test_str_replace_editor_view_range_rejects_non_integers(monkeypatch): + actions_mod = _load_actions_module(monkeypatch) + + with pytest.raises( + actions_mod.FormatError, match="view_range values must be integers" + ): + actions_mod.parse_toolcall_actions( + [ + _tool_call( + "str_replace_editor", + { + "command": "view", + "path": "/testbed/pkg/file.py", + "view_range": ["one", 4], + }, + ) + ], + format_error_template="{{ error }}", + ) diff --git a/tests/unit/evaluation/test_swe_bench_scorer.py b/tests/unit/evaluation/test_swe_bench_scorer.py new file mode 100644 index 000000000..b1c47c332 --- /dev/null +++ b/tests/unit/evaluation/test_swe_bench_scorer.py @@ -0,0 +1,551 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Unit tests for service-backed SWEBenchScorer.""" + +import json +from pathlib import Path +from unittest.mock import MagicMock + +import msgspec +import pandas as pd +import pytest +import yaml + +# isort: split +from inference_endpoint.evaluation import scoring as scoring_mod +from inference_endpoint.evaluation.scoring import Scorer, SWEBenchScorer +from inference_endpoint.exceptions import SetupError + +pytestmark = pytest.mark.unit + +_DATASET_NAME = "swe_bench" +_MODEL_NAME = "TestOrg/test-model-7b" + + +class FakeTqdm: + instances: list["FakeTqdm"] = [] + + def __init__(self, *, total, desc, unit): + self.total = total + self.desc = desc + self.unit = unit + self.n = 0 + self.closed = False + self.refreshes = 0 + self.updates: list[int] = [] + type(self).instances.append(self) + + def update(self, n): + self.updates.append(n) + self.n += n + + def refresh(self): + self.refreshes += 1 + + def close(self): + self.closed = True + + +def _write_benchmark_config( + report_dir: Path, + model_params: dict | None = None, + endpoint_urls: list[str] | None = None, +) -> None: + mp: dict = {"name": _MODEL_NAME} + if model_params is not None: + mp.update(model_params) + (report_dir / "config.yaml").write_text( + yaml.dump( + { + "model_params": mp, + "endpoint_config": { + "endpoints": endpoint_urls or ["http://endpoint-host:30000"], + "api_key": "secret-key", + }, + } + ) + ) + + +def _write_sample_idx_map(report_dir: Path, n: int = 3) -> None: + idx_map = {_DATASET_NAME: {f"uuid-{i}": i for i in range(n)}} + (report_dir / "sample_idx_map.json").write_bytes(msgspec.json.encode(idx_map)) + + +def _make_dataset(n: int = 3) -> MagicMock: + ds = MagicMock() + ds.dataframe = pd.DataFrame( + { + "instance_id": [f"repo__repo-{i}" for i in range(n)], + "prompt": ["placeholder"] * n, + } + ) + ds.num_samples.return_value = n + return ds + + +def _make_scorer(report_dir: Path, *, dataset=None, **kwargs) -> SWEBenchScorer: + return SWEBenchScorer( + dataset_name=_DATASET_NAME, + dataset=dataset or _make_dataset(), + report_dir=report_dir, + swebench_service_url="http://service-host:18080", + poll_interval_s=0.01, + **kwargs, + ) + + +@pytest.fixture +def report_dir(tmp_path: Path) -> Path: + d = tmp_path / "report" + d.mkdir() + _write_benchmark_config(d) + _write_sample_idx_map(d) + return d + + +class TestSWEBenchScorerRegistration: + def test_registered(self): + assert "swe_bench_scorer" in Scorer.PREDEFINED + assert Scorer.get("swe_bench_scorer") is SWEBenchScorer + + def test_skip_endpoint_phase(self): + assert SWEBenchScorer.SKIP_ENDPOINT_PHASE is True + + def test_external_sample_count(self): + assert SWEBenchScorer.external_sample_count({"num_instances": 50}) == 50 + assert ( + SWEBenchScorer.external_sample_count({}) + == SWEBenchScorer.DEFAULT_NUM_INSTANCES + ) + assert SWEBenchScorer.external_sample_count({"num_instances": "bad"}) is None + assert SWEBenchScorer.external_sample_count({"num_instances": 0}) is None + + def test_poll_interval_defaults_when_constructor_passes_none(self): + options = SWEBenchScorer._resolve_options( + { + "swebench_service_url": "http://service-host:18080", + "poll_interval_s": None, + } + ) + + assert options["poll_interval_s"] == SWEBenchScorer.DEFAULT_POLL_INTERVAL_S + + +class TestSWEBenchScorerPreflight: + def test_preflight_calls_health(self, monkeypatch): + calls: list[tuple[str, str, str | None]] = [] + + def fake_http_json(url, *, method="GET", **kwargs): + calls.append((url, method, kwargs.get("auth_token"))) + return { + "api_version": "v1", + "capabilities": [ + "swebench.run", + "swebench.cancel", + "artifacts.download", + ], + } + + monkeypatch.setattr(SWEBenchScorer, "_http_json", fake_http_json) + + SWEBenchScorer.preflight( + { + "swebench_service_url": "http://service-host:18080", + "swebench_service_auth_token": "tok", + "subset": "lite", + "num_instances": 1, + } + ) + + assert calls == [("http://service-host:18080/health", "GET", "tok")] + + def test_preflight_never_calls_docker_or_subprocess(self, monkeypatch): + def fail_run(*args, **kwargs): + pytest.fail("client preflight must not run local subprocesses") + + monkeypatch.setattr(scoring_mod.subprocess, "run", fail_run) + monkeypatch.setattr( + SWEBenchScorer, + "_http_json", + classmethod( + lambda cls, url, **kwargs: { + "api_version": "v1", + "capabilities": [ + "swebench.run", + "swebench.cancel", + "artifacts.download", + ], + } + ), + ) + + SWEBenchScorer.preflight({"swebench_service_url": "http://service-host:18080"}) + + def test_missing_service_url_raises(self): + with pytest.raises(SetupError, match="swebench_service_url is required"): + SWEBenchScorer.preflight({}) + + def test_capability_mismatch_raises(self, monkeypatch): + monkeypatch.setattr( + SWEBenchScorer, + "_http_json", + classmethod( + lambda cls, url, **kwargs: {"api_version": "v1", "capabilities": []} + ), + ) + + with pytest.raises(SetupError, match="missing required capabilities"): + SWEBenchScorer.preflight({"swebench_service_url": "http://service-host"}) + + def test_api_version_mismatch_raises(self, monkeypatch): + monkeypatch.setattr( + SWEBenchScorer, + "_http_json", + classmethod( + lambda cls, url, **kwargs: { + "api_version": "v2", + "capabilities": [ + "swebench.run", + "swebench.cancel", + "artifacts.download", + ], + } + ), + ) + + with pytest.raises(SetupError, match="API version mismatch"): + SWEBenchScorer.preflight({"swebench_service_url": "http://service-host"}) + + +class TestSWEBenchScorerScore: + def test_score_submits_exact_instance_ids_and_computes_rate( + self, report_dir, monkeypatch + ): + payloads: list[dict] = [] + + def fake_http_json(url, *, method="GET", payload=None, **kwargs): + if method == "POST": + payloads.append(payload) + return { + "run_id": "run-1", + "status": "succeeded", + "result": {"resolved_instances": 2, "submitted_instances": 3}, + "artifacts": [], + } + raise AssertionError(f"unexpected GET {url}") + + monkeypatch.setattr(SWEBenchScorer, "_http_json", fake_http_json) + scorer = _make_scorer(report_dir) + + score, n_repeats = scorer.score() + + assert score == pytest.approx(2 / 3) + assert n_repeats == 1 + assert scorer.complete is True + assert payloads[0]["evaluated_instance_ids"] == [ + "repo__repo-0", + "repo__repo-1", + "repo__repo-2", + ] + assert "benchmark_config" not in payloads[0] + assert payloads[0]["endpoint_urls"] == ["http://endpoint-host:30000"] + assert payloads[0]["endpoint_api_key"] == "secret-key" + assert payloads[0]["generation_params"] == {} + assert payloads[0]["template"] == "default" + assert (report_dir / "swe_bench_results.json").exists() + + def test_score_polls_until_terminal(self, report_dir, monkeypatch): + calls: list[str] = [] + + def fake_http_json(url, *, method="GET", payload=None, **kwargs): + calls.append(url) + if method == "POST": + return {"run_id": "run-1", "status": "queued"} + return { + "run_id": "run-1", + "status": "succeeded", + "result": {"resolved_instances": 1, "submitted_instances": 3}, + "artifacts": [], + } + + monkeypatch.setattr(SWEBenchScorer, "_http_json", fake_http_json) + score, _ = _make_scorer(report_dir).score() + + assert score == pytest.approx(1 / 3) + assert calls[-1] == "http://service-host:18080/v1/runs/run-1" + + def test_score_forwards_auth_token_to_service_calls(self, report_dir, monkeypatch): + calls: list[tuple[str, str, str | None]] = [] + downloads: list[tuple[str, str | None]] = [] + phase = "success" + + def fake_http_json( + url, *, method="GET", payload=None, auth_token=None, **kwargs + ): + calls.append((url, method, auth_token)) + if phase == "success": + if method == "POST": + return {"run_id": "run-1", "status": "running"} + return { + "run_id": "run-1", + "status": "succeeded", + "result": {"resolved_instances": 1, "submitted_instances": 3}, + "artifacts": [ + { + "name": "preds.json", + "url": "/v1/runs/run-1/artifacts/preds.json", + } + ], + } + if method == "POST" and url.endswith("/cancel"): + return {"run_id": "run-2", "status": "cancelled"} + if method == "POST": + return {"run_id": "run-2", "status": "running"} + raise SetupError("poll failed") + + def fake_download(service_url, status, output_dir, auth_token=None): + downloads.append((status["run_id"], auth_token)) + + monkeypatch.setattr(SWEBenchScorer, "_http_json", fake_http_json) + monkeypatch.setattr(SWEBenchScorer, "_download_artifacts", fake_download) + + scorer = _make_scorer(report_dir, swebench_service_auth_token="tok") + score, _ = scorer.score() + assert score == pytest.approx(1 / 3) + + phase = "failure" + failed_scorer = _make_scorer(report_dir, swebench_service_auth_token="tok") + assert failed_scorer.score() == (None, 1) + + assert all(auth_token == "tok" for _, _, auth_token in calls) + assert ("run-1", "tok") in downloads + assert ( + "http://service-host:18080/v1/runs/run-2/cancel", + "POST", + "tok", + ) in calls + + def test_score_rejects_multiple_endpoint_urls(self, report_dir, monkeypatch): + _write_benchmark_config( + report_dir, + endpoint_urls=["http://endpoint-a:30000", "http://endpoint-b:30000"], + ) + monkeypatch.setattr(SWEBenchScorer, "_http_json", MagicMock()) + scorer = _make_scorer(report_dir) + + with pytest.raises(SetupError, match="exactly one endpoint URL"): + scorer.score() + + SWEBenchScorer._http_json.assert_not_called() + + def test_score_renders_agent_and_eval_progress_bars(self, report_dir, monkeypatch): + FakeTqdm.instances = [] + statuses = [ + { + "run_id": "run-1", + "status": "running", + "phase": "agent", + "agent_total": 3, + "agent_completed": 1, + "eval_total": 0, + "eval_completed": 0, + }, + { + "run_id": "run-1", + "status": "running", + "phase": "agent", + "agent_total": 3, + "agent_completed": 1, + "eval_total": 0, + "eval_completed": 0, + }, + { + "run_id": "run-1", + "status": "running", + "phase": "eval", + "agent_total": 3, + "agent_completed": 3, + "eval_total": 3, + "eval_completed": 2, + }, + { + "run_id": "run-1", + "status": "succeeded", + "phase": "succeeded", + "agent_total": 3, + "agent_completed": 3, + "eval_total": 3, + "eval_completed": 3, + "result": {"resolved_instances": 2, "submitted_instances": 3}, + "artifacts": [], + }, + ] + + def fake_http_json(url, *, method="GET", payload=None, **kwargs): + return statuses.pop(0) + + monkeypatch.setattr(scoring_mod, "tqdm", FakeTqdm) + monkeypatch.setattr(SWEBenchScorer, "_http_json", fake_http_json) + + score, _ = _make_scorer(report_dir).score() + + assert score == pytest.approx(2 / 3) + assert [bar.desc for bar in FakeTqdm.instances] == [ + "SWE-bench agent", + "SWE-bench eval", + ] + assert FakeTqdm.instances[0].n == 3 + assert FakeTqdm.instances[1].n == 3 + assert all(bar.closed for bar in FakeTqdm.instances) + + def test_score_without_progress_does_not_open_tqdm(self, report_dir, monkeypatch): + statuses = [ + {"run_id": "run-1", "status": "running"}, + { + "run_id": "run-1", + "status": "succeeded", + "result": {"resolved_instances": 1, "submitted_instances": 3}, + "artifacts": [], + }, + ] + + def fake_tqdm(*args, **kwargs): + pytest.fail("progress bar should not open without progress fields") + + def fake_http_json(url, *, method="GET", payload=None, **kwargs): + return statuses.pop(0) + + monkeypatch.setattr(scoring_mod, "tqdm", fake_tqdm) + monkeypatch.setattr(SWEBenchScorer, "_http_json", fake_http_json) + + score, _ = _make_scorer(report_dir).score() + + assert score == pytest.approx(1 / 3) + + def test_interrupt_cancels_service_run(self, report_dir, monkeypatch): + calls: list[tuple[str, str]] = [] + + def fake_http_json(url, *, method="GET", payload=None, **kwargs): + calls.append((url, method)) + if method == "POST" and url == "http://service-host:18080/v1/runs": + return {"run_id": "run-1", "status": "running"} + if ( + method == "POST" + and url == "http://service-host:18080/v1/runs/run-1/cancel" + ): + return {"run_id": "run-1", "status": "cancelled"} + raise KeyboardInterrupt + + monkeypatch.setattr(SWEBenchScorer, "_http_json", fake_http_json) + scorer = _make_scorer(report_dir) + + with pytest.raises(KeyboardInterrupt): + scorer.score() + + assert ("http://service-host:18080/v1/runs/run-1/cancel", "POST") in calls + + def test_service_failure_marks_incomplete(self, report_dir, monkeypatch): + monkeypatch.setattr( + SWEBenchScorer, + "_http_json", + classmethod( + lambda cls, url, **kwargs: { + "run_id": "run-1", + "status": "failed", + "error": "boom", + } + ), + ) + scorer = _make_scorer(report_dir) + + score, n_repeats = scorer.score() + + assert score is None + assert n_repeats == 1 + assert scorer.complete is False + + def test_missing_result_marks_incomplete(self, report_dir, monkeypatch): + monkeypatch.setattr( + SWEBenchScorer, + "_http_json", + classmethod( + lambda cls, url, **kwargs: {"run_id": "run-1", "status": "succeeded"} + ), + ) + scorer = _make_scorer(report_dir) + + score, _ = scorer.score() + + assert score is None + assert scorer.complete is False + + def test_zero_denominator_marks_incomplete(self, report_dir, monkeypatch): + monkeypatch.setattr(SWEBenchScorer, "_http_json", MagicMock()) + scorer = _make_scorer(report_dir, dataset=_make_dataset(n=0)) + + score, _ = scorer.score() + + assert score is None + assert scorer.complete is False + SWEBenchScorer._http_json.assert_not_called() + + def test_partial_submission_marks_incomplete(self, report_dir, monkeypatch): + def fake_http_json(url, *, method="GET", payload=None, **kwargs): + return { + "run_id": "run-1", + "status": "succeeded", + "result": {"resolved_instances": 1, "submitted_instances": 2}, + "artifacts": [], + } + + monkeypatch.setattr(SWEBenchScorer, "_http_json", fake_http_json) + scorer = _make_scorer(report_dir) + + score, _ = scorer.score() + + assert score == pytest.approx(1 / 3) + assert scorer.complete is False + + def test_unsafe_artifact_url_is_not_downloaded(self, report_dir, monkeypatch): + def fail_urlopen(*args, **kwargs): + pytest.fail("unsafe artifact URL must not be fetched") + + monkeypatch.setattr(scoring_mod.urllib_request, "urlopen", fail_urlopen) + + SWEBenchScorer._download_artifact( + "http://service-host:18080/", + {"name": "preds.json", "url": "http://evil/preds.json"}, + report_dir, + "run-1", + ) + + assert not (report_dir / "preds.json").exists() + + def test_artifact_result_fallback(self, report_dir, monkeypatch): + result_path = report_dir / "swe_bench_results.json" + + def fake_http_json(url, *, method="GET", payload=None, **kwargs): + return { + "run_id": "run-1", + "status": "succeeded", + "artifacts": [ + { + "name": "swe_bench_results.json", + "url": "/v1/runs/run-1/artifacts/swe_bench_results.json", + } + ], + } + + def fake_download(service_url, status, output_dir, auth_token=None): + result_path.write_text( + json.dumps({"resolved_instances": 1, "submitted_instances": 3}) + ) + + monkeypatch.setattr(SWEBenchScorer, "_http_json", fake_http_json) + monkeypatch.setattr(SWEBenchScorer, "_download_artifacts", fake_download) + + score, _ = _make_scorer(report_dir).score() + + assert score == pytest.approx(1 / 3) diff --git a/uv.lock b/uv.lock index a805b0a17..0a11069fe 100644 --- a/uv.lock +++ b/uv.lock @@ -1324,6 +1324,7 @@ dependencies = [ { name = "pydantic", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'x86_64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'darwin' and sys_platform != 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-dev') or (sys_platform != 'darwin' and sys_platform != 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-performance') or (sys_platform != 'darwin' and sys_platform != 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-test') or (sys_platform == 'darwin' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-dev') or (sys_platform == 'darwin' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-performance') or (sys_platform == 'darwin' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-test') or (sys_platform == 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-dev') or (sys_platform == 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-performance') or (sys_platform == 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-test')" }, { name = "pydantic-core", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'x86_64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'darwin' and sys_platform != 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-dev') or (sys_platform != 'darwin' and sys_platform != 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-performance') or (sys_platform != 'darwin' and sys_platform != 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-test') or (sys_platform == 'darwin' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-dev') or (sys_platform == 'darwin' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-performance') or (sys_platform == 'darwin' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-test') or (sys_platform == 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-dev') or (sys_platform == 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-performance') or (sys_platform == 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-test')" }, { name = "pytz", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'x86_64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'darwin' and sys_platform != 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-dev') or (sys_platform != 'darwin' and sys_platform != 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-performance') or (sys_platform != 'darwin' and sys_platform != 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-test') or (sys_platform == 'darwin' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-dev') or (sys_platform == 'darwin' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-performance') or (sys_platform == 'darwin' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-test') or (sys_platform == 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-dev') or (sys_platform == 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-performance') or (sys_platform == 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-test')" }, + { name = "pyyaml", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'x86_64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'darwin' and sys_platform != 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-dev') or (sys_platform != 'darwin' and sys_platform != 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-performance') or (sys_platform != 'darwin' and sys_platform != 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-test') or (sys_platform == 'darwin' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-dev') or (sys_platform == 'darwin' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-performance') or (sys_platform == 'darwin' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-test') or (sys_platform == 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-dev') or (sys_platform == 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-performance') or (sys_platform == 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-test')" }, { name = "pyzmq", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'x86_64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'darwin' and sys_platform != 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-dev') or (sys_platform != 'darwin' and sys_platform != 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-performance') or (sys_platform != 'darwin' and sys_platform != 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-test') or (sys_platform == 'darwin' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-dev') or (sys_platform == 'darwin' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-performance') or (sys_platform == 'darwin' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-test') or (sys_platform == 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-dev') or (sys_platform == 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-performance') or (sys_platform == 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-test')" }, { name = "rich", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'x86_64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'darwin' and sys_platform != 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-dev') or (sys_platform != 'darwin' and sys_platform != 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-performance') or (sys_platform != 'darwin' and sys_platform != 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-test') or (sys_platform == 'darwin' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-dev') or (sys_platform == 'darwin' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-performance') or (sys_platform == 'darwin' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-test') or (sys_platform == 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-dev') or (sys_platform == 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-performance') or (sys_platform == 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-test')" }, { name = "sentencepiece", marker = "(platform_machine == 'arm64' and sys_platform == 'darwin') or (platform_machine == 'x86_64' and sys_platform == 'darwin') or (platform_machine == 'aarch64' and sys_platform == 'linux') or (platform_machine == 'x86_64' and sys_platform == 'linux') or (sys_platform != 'darwin' and sys_platform != 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-dev') or (sys_platform != 'darwin' and sys_platform != 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-performance') or (sys_platform != 'darwin' and sys_platform != 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-test') or (sys_platform == 'darwin' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-dev') or (sys_platform == 'darwin' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-performance') or (sys_platform == 'darwin' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-test') or (sys_platform == 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-dev') or (sys_platform == 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-performance') or (sys_platform == 'linux' and extra == 'extra-18-inference-endpoint-bfcl' and extra == 'extra-18-inference-endpoint-test')" }, @@ -1418,6 +1419,7 @@ requires-dist = [ { name = "pytest-timeout", marker = "extra == 'test'", specifier = "==2.4.0" }, { name = "pytest-xdist", marker = "extra == 'test'", specifier = "==3.8.0" }, { name = "pytz", specifier = "==2026.1.post1" }, + { name = "pyyaml", specifier = "==6.0.3" }, { name = "pyzmq", specifier = "==27.1.0" }, { name = "rich", specifier = "==14.3.3" }, { name = "ruff", marker = "extra == 'dev'", specifier = "==0.15.8" },