test(snapshot): Add structured JSON fixture targets#338
Merged
cameroncooke merged 9 commits intomainfrom Apr 14, 2026
Merged
Conversation
Add a structured JSON fixture corpus for the planned CLI json output mode. Define shared envelope and schema-family targets in docs/dev/STRUCTURED_JSON_OUTPUT_PLAN.md and capture the expected payloads across simulator, device, macOS, SwiftPM, debugging, discovery, and session workflows. This keeps the current producer unchanged while giving the upcoming implementation a concrete fixture contract to build against. Co-Authored-By: OpenAI Codex <noreply@openai.com>
commit: |
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Device-list schema missing
isAvailablefield in resources fixture- Added the missing isAvailable field to all device entries in resources/devices--success.json to match the device-list schema contract.
- ✅ Fixed: Simulator success fixtures inconsistently omit
diagnosticsfield- Added the missing diagnostics field with empty warnings and errors arrays to simulator success fixtures for stop, install, and launch-app operations.
Or push these changes by commenting:
@cursor push d6600841ee
Preview (d6600841ee)
diff --git a/src/snapshot-tests/__fixtures__/json/resources/devices--success.json b/src/snapshot-tests/__fixtures__/json/resources/devices--success.json
--- a/src/snapshot-tests/__fixtures__/json/resources/devices--success.json
+++ b/src/snapshot-tests/__fixtures__/json/resources/devices--success.json
@@ -6,32 +6,36 @@
"data": {
"devices": [
{
- "name": "Cameron’s iPhone 16 Pro Max",
+ "name": "Cameron's iPhone 16 Pro Max",
"deviceId": "<UUID>",
"platform": "iOS",
"osVersion": "26.3.1 (a)",
- "state": "<STATUS>"
+ "state": "<STATUS>",
+ "isAvailable": true
},
{
"name": "iPhone",
"deviceId": "<UUID>",
"platform": "iOS",
"osVersion": "26.1",
- "state": "<STATUS>"
+ "state": "<STATUS>",
+ "isAvailable": false
},
{
- "name": "Cameron’s Apple Watch",
+ "name": "Cameron's Apple Watch",
"deviceId": "<UUID>",
"platform": "watchOS",
"osVersion": "10.6.1",
- "state": "<STATUS>"
+ "state": "<STATUS>",
+ "isAvailable": false
},
{
- "name": "Cameron’s Apple Watch",
+ "name": "Cameron's Apple Watch",
"deviceId": "<UUID>",
"platform": "watchOS",
"osVersion": "26.3",
- "state": "<STATUS>"
+ "state": "<STATUS>",
+ "isAvailable": true
}
]
}
diff --git a/src/snapshot-tests/__fixtures__/json/simulator/install--success.json b/src/snapshot-tests/__fixtures__/json/simulator/install--success.json
--- a/src/snapshot-tests/__fixtures__/json/simulator/install--success.json
+++ b/src/snapshot-tests/__fixtures__/json/simulator/install--success.json
@@ -10,6 +10,10 @@
"artifacts": {
"simulatorId": "<UUID>",
"appPath": "<HOME>/Library/Developer/XcodeBuildMCP/DerivedData/Build/Products/Debug-iphonesimulator/CalculatorApp.app"
+ },
+ "diagnostics": {
+ "warnings": [],
+ "errors": []
}
}
}
diff --git a/src/snapshot-tests/__fixtures__/json/simulator/launch-app--success.json b/src/snapshot-tests/__fixtures__/json/simulator/launch-app--success.json
--- a/src/snapshot-tests/__fixtures__/json/simulator/launch-app--success.json
+++ b/src/snapshot-tests/__fixtures__/json/simulator/launch-app--success.json
@@ -13,6 +13,10 @@
"processId": "<PID>",
"runtimeLogPath": "<HOME>/Library/Developer/XcodeBuildMCP/logs/io.sentry.calculatorapp_<TIMESTAMP>_pid<PID>.log",
"osLogPath": "<HOME>/Library/Developer/XcodeBuildMCP/logs/io.sentry.calculatorapp_oslog_<TIMESTAMP>_pid<PID>.log"
+ },
+ "diagnostics": {
+ "warnings": [],
+ "errors": []
}
}
}
diff --git a/src/snapshot-tests/__fixtures__/json/simulator/stop--success.json b/src/snapshot-tests/__fixtures__/json/simulator/stop--success.json
--- a/src/snapshot-tests/__fixtures__/json/simulator/stop--success.json
+++ b/src/snapshot-tests/__fixtures__/json/simulator/stop--success.json
@@ -10,6 +10,10 @@
"artifacts": {
"simulatorId": "<UUID>",
"bundleId": "io.sentry.calculatorapp"
+ },
+ "diagnostics": {
+ "warnings": [],
+ "errors": []
}
}
}This Bugbot Autofix run was free. To enable autofix for future PRs, go to the Cursor dashboard.
src/snapshot-tests/__fixtures__/json/resources/devices--success.json
Outdated
Show resolved
Hide resolved
Collaborator
Author
- Add missing isAvailable field to device-list schema in resources/devices--success.json - Add missing diagnostics field to simulator success fixtures (stop, install, launch-app) Applied via @cursor push command
src/snapshot-tests/__fixtures__/json/simulator/test--failure.json
Outdated
Show resolved
Hide resolved
src/snapshot-tests/__fixtures__/json/simulator/test--failure.json
Outdated
Show resolved
Hide resolved
src/snapshot-tests/__fixtures__/json/resources/devices--success.json
Outdated
Show resolved
Hide resolved
…form schema contract - Add missing summary.durationMs and summary.target to all simulator fixtures - Fix singular "Test failed" to "Tests failed" in simulator failure fixtures - Normalize counts key ordering to passed/failed/skipped in simulator test failure - Align resources/devices field ordering with device/list fixture
src/snapshot-tests/__fixtures__/json/simulator/build--error-wrong-scheme.json
Show resolved
Hide resolved
src/snapshot-tests/__fixtures__/json/macos/test--error-wrong-scheme.json
Outdated
Show resolved
Hide resolved
src/snapshot-tests/__fixtures__/json/swift-package/stop--error-no-process.json
Outdated
Show resolved
Hide resolved
…cross all fixtures - durationMs: "<DURATION>" -> 1234 - processId/pid: "<PID>" -> 99999 - Test failure counts: "<PASS_COUNT>"/"<FAIL_COUNT>"/"<SKIP_COUNT>" -> mock integers matching discovered totals - session-status: uptimeMs/rssBytes/heapUsedBytes -> mock integers - doctor: manifest counts, simulatorVideoCaptureSupported -> proper types - stack: threadId/index -> integers - swift-package/list: rename uptime -> uptimeSeconds - Fix remaining "Test failed" -> "Tests failed" in macos and swift-package error fixtures
…drop resource fixtures - Convert debug variables scopes from array to keyed object, remove displayName - Remove output.stdout/stderr from build-and-run success fixtures - Remove remainingCount from test failure discovered items - Delete resource fixtures (devices, doctor, session-status, simulators)
Add a dedicated schema corpus for structured JSON fixtures and validate each discovered fixture against its declared schema and version. This gives the hand-crafted fixture contracts an executable check without coupling them to the existing text snapshot harness. Co-Authored-By: OpenAI Codex <noreply@openai.com>
Document schema versioning and deployment policy and add a workflow that syncs published schema files into the website repository for Vercel deployment. This keeps schema authoring in this repository while making the published schema URLs durable and real. Co-Authored-By: OpenAI Codex <noreply@openai.com>
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 102658c. Configure here.
Limit the publish workflow token to read-only contents access and avoid interpolating the deploy key secret directly inside the shell guard. This keeps the workflow aligned with GitHub Actions security expectations and removes a brittle secret-handling pattern. Co-Authored-By: OpenAI Codex <noreply@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Add a structured JSON fixture corpus for the planned CLI
--output jsonmode.This introduces a target fixture tree under
src/snapshot-tests/__fixtures__/jsonand a companion planning doc indocs/dev/STRUCTURED_JSON_OUTPUT_PLAN.md. The fixtures are normalized around shared schema families so we can review the output contract before wiring the producer and snapshot harness to it.The main goal here is to lock down the result shapes early: text and current event-stream output stay unchanged for now, while the future structured JSON mode gets a concrete, reviewable contract across simulator, device, macOS, SwiftPM, debugging, discovery, session-management, and UI automation workflows.
I kept this PR fixture-only. It does not update the runtime producer, tests, or harness plumbing yet.