Add machine-readable scenario summary report

[shreyaskommuri]

Summary

• Adds a default summary scenario reporter that writes cloudai-summary.json next to the existing scenario reports.
• Includes scenario name/status, system metadata, result directory, scenario-level report artifacts, per-test-run status, output paths, artifacts, and metrics already exposed through configured metric reporters.
• Registers the reporter by default and keeps tarball as the final scenario reporter.

Fixes #917.

Test Plan

Environment:

• macOS
• Python 3.10.20 via uv run

Commands and results:

$ uv run ruff check src/cloudai/reporter.py src/cloudai/registration.py src/cloudai/_core/registry.py src/cloudai/core.py tests/test_reporter.py tests/test_init.py
All checks passed!

$ uv run ruff format --check src/cloudai/reporter.py src/cloudai/registration.py src/cloudai/_core/registry.py src/cloudai/core.py tests/test_reporter.py tests/test_init.py
6 files already formatted

$ uv run pytest tests/test_reporter.py tests/test_init.py tests/test_registry.py
60 passed in 0.43s

$ uv run pytest
1573 passed, 4 skipped, 463 deselected in 5.14s

Additional Notes

• This keeps existing HTML, tarball, comparison, and workload-specific reports unchanged.
• The summary intentionally uses metrics CloudAI already exposes through each test run's configured metric reporter, so this PR adds a stable automation artifact without introducing new workload-specific parsing logic.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 03018a0f-256f-4c8f-a6c2-b673f3d430cf

📥 Commits

Reviewing files that changed from the base of the PR and between dfa28da and 008e5d4.

📒 Files selected for processing (2)

• src/cloudai/registration.py
• tests/test_init.py

---

📝 Walkthrough

Walkthrough

Adds SummaryReporter to generate cloudai-summary.json, registers and exports it, updates report ordering to place summary before tarball, and extends tests for registry ordering and JSON summary contents.

Changes

Summary reporter and registry integration

Layer / File(s)	Summary
SummaryReporter implementation src/cloudai/reporter.py	Adds SummaryReporter to generate cloudai-summary.json with scenario metadata, artifact discovery, per-test-run summaries, DSE sweep nesting, and metric filtering that skips METRIC_ERROR.
Public export and registration src/cloudai/core.py, src/cloudai/registration.py, src/cloudai/_core/registry.py	Exports SummaryReporter from cloudai.core, registers an enabled "summary" scenario report, and updates scenario report ordering so summary sorts before tarball.
Registry and summary output tests tests/test_init.py, tests/test_reporter.py	Updates registry expectations to include summary, and adds a JSON test that runs SummaryReporter.generate() and asserts the emitted summary structure, artifacts, and metrics.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐰 I hopped through JSON, neat and bright,
With summary trails in morning light.
A tarball last, a summary first,
For tools to sip when files are nursed.

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly matches the main change: adding a machine-readable scenario summary report.
Description check	✅ Passed	The description directly describes adding the summary reporter and its JSON artifact.
Linked Issues check	✅ Passed	The changes implement the requested stable machine-readable summary artifact and include the core fields from #917.
Out of Scope Changes check	✅ Passed	The diff stays focused on the summary reporter, registration, exports, and related tests.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[podkidyshev]

this won't work gracefully when we run sweeping. in the sweeping case, the folders structure will be

/scenario foler/
               /case 1/
                      /sweep 1/
                              /some-report.html
                      /sweep 2/
                              /some-report.html

[podkidyshev]

sweeping is actually quite important in our work. I believe this json summary report should bring insights if those were run

[shreyaskommuri]

Addressed the requested changes and updated the PR branch from latest main.

What changed:

• Replaced the summary reporter regression with a full-object JSON assert.
• Trimmed the first summary contract to the minimum useful automation fields: scenario, status, result_dir, reports, and test_runs.
• Grouped DSE/sweep outputs under their parent test run via a sweeps list, including case-level artifacts such as trajectory.csv and per-sweep nested report artifacts.

Validation on the updated PR head dfa28da:

• uv run ruff check src/cloudai/reporter.py tests/test_reporter.py tests/test_init.py src/cloudai/registration.py src/cloudai/_core/registry.py src/cloudai/core.py -> passed
• uv run ruff format --check src/cloudai/reporter.py tests/test_reporter.py tests/test_init.py src/cloudai/registration.py src/cloudai/_core/registry.py src/cloudai/core.py -> passed
• uv run pytest tests/test_reporter.py tests/test_init.py tests/test_registry.py -q -> 60 passed
• uv run pytest -q -> 1599 passed, 4 skipped, 476 deselected