Skip to content

Report render comparison artifacts from CI#876

Open
johnml1135 wants to merge 7 commits into
mainfrom
better_render_ci
Open

Report render comparison artifacts from CI#876
johnml1135 wants to merge 7 commits into
mainfrom
better_render_ci

Conversation

@johnml1135
Copy link
Copy Markdown
Contributor

@johnml1135 johnml1135 commented May 11, 2026

Adds CI support for render snapshot failure artifacts.

  • Collects render verifier .received.png, .verified.png, .diff.png, and metadata into Output/RenderArtifacts
  • Uploads a run artifact with index.html, README.md, and manifest.json
  • Adds a workflow summary link to the artifact
  • Updates a single PR comment with artifact links for failing render runs, and marks it resolved after a successful run

Local verification:

  • ./Build/Agent/Test-CollectRenderArtifacts.ps1

This change is Reviewable

Copilot AI review requested due to automatic review settings May 11, 2026 17:01
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds CI support for collecting and surfacing render snapshot comparison artifacts when render verification fails, including downloadable reports and PR-visible links.

Changes:

  • Adds PowerShell tooling to collect .received.png/.verified.png/.diff.png (+ metadata) into a structured artifact with index.html, README.md, and manifest.json.
  • Updates CI workflow to upload the artifact, add a summary link, and maintain a single PR comment with run/artifact links.
  • Adds a local smoke-test script for the artifact collector.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

File Description
Src/Common/RootSite/RootSiteTests/RenderBenchmarkTestsBase.cs Modifies sample render text (currently causes intentional render diffs).
Build/Agent/Test-CollectRenderArtifacts.ps1 Adds a local smoke test harness for the artifact collection script.
Build/Agent/Collect-RenderArtifacts.ps1 New artifact collector + report/manifest generation and GitHub output wiring.
.github/workflows/CI.yml Uploads the artifact on failure and posts/updates a PR comment + workflow summary.

Comment thread Src/Common/RootSite/RootSiteTests/RenderBenchmarkTestsBase.cs Outdated
Comment thread Build/Agent/Collect-RenderArtifacts.ps1 Outdated
Comment thread .github/workflows/CI.yml Outdated
Comment thread Build/Agent/Collect-RenderArtifacts.ps1 Outdated
Comment thread Build/Agent/Collect-RenderArtifacts.ps1 Outdated
Comment thread Build/Agent/Collect-RenderArtifacts.ps1 Outdated
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 11, 2026

Render comparison artifacts

No render snapshot failures were captured in the latest successful CI run.

Latest run: b1fdb75c831e run 25688716247.1.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 11, 2026

NUnit Tests

    1 files  ±0      1 suites  ±0   11m 19s ⏱️ -12s
4 187 tests ±0  4 116 ✅ ±0  71 💤 ±0  0 ❌ ±0 
4 196 runs  ±0  4 125 ✅ ±0  71 💤 ±0  0 ❌ ±0 

Results for commit 789cdfc. ± Comparison against base commit 502b832.

♻️ This comment has been updated with latest results.

Copy link
Copy Markdown
Contributor

@jasonleenaylor jasonleenaylor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you investigate if reg-actions can handle the reporting when we have detected the diffs?

@jasonleenaylor reviewed 1 file and all commit messages, and made 1 comment.
Reviewable status: 1 of 5 files reviewed, all discussions resolved.

@johnml1135
Copy link
Copy Markdown
Contributor Author

Decision note for this PR:

I kept the current render-artifact collection flow and only swapped the sticky PR comment maintenance to marocchino/sticky-pull-request-comment.

What I considered:

  • reg-viz/reg-actions

    • Good when it owns the visual-regression lifecycle: collect current screenshots, fetch/reference a baseline, run image comparison, then publish a report/comment.
    • Not a good fit for this repo right now because our tests already produce Verify-style *.received.png, *.verified.png, *.diff.png, plus index.html/manifest.json as the reporting payload.
    • Using reg-actions here would mean restructuring the pipeline so it compares raw screenshots itself and becomes the source of truth for comparison results. That would duplicate or replace the comparison work we already do.
  • Argos

    • Stronger than reg-actions if we want a full managed visual-regression platform.
    • Not appropriate right now because it wants us to adopt its reporter/upload/baseline/review workflow and external review UI.
    • We are not ready to move baseline ownership or render-review flow out of the current repo/test pipeline.
  • Generic report actions (publish-report, unit-test-report actions)

    • Useful for JUnit/TRX-style structured test results.
    • They do not understand our image-diff artifact bundle, so they would not replace the collector/report format we already have.

Why this change:

marocchino/sticky-pull-request-comment gives us the one prebuilt capability we actually need now: stable PR comment lifecycle management. That lets us keep the existing Output/RenderArtifacts bundle, index.html, manifest.json, and step summary while removing the custom GitHub comment API handling from CI.

So the current direction is:

  • keep the existing render artifact generation,
  • keep the existing artifact upload,
  • keep the existing step summary,
  • use a standard sticky-comment action only for the PR comment update.

@github-actions
Copy link
Copy Markdown

Render comparison artifacts

No render snapshot failures were captured in the latest successful CI run.

Latest run: c38dcfe86533 run 25742517199.1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants