Skip to content

feat: add gesture command coverage#576

Open
thymikee wants to merge 4 commits into
mainfrom
codex/gesture-commands-testbed
Open

feat: add gesture command coverage#576
thymikee wants to merge 4 commits into
mainfrom
codex/gesture-commands-testbed

Conversation

@thymikee
Copy link
Copy Markdown
Member

@thymikee thymikee commented May 21, 2026

Summary

Adds end-to-end gesture support and a React Native Gesture Handler test bed for pan, pinch, rotate, and fling gestures.

Details:

  • Uses the canonical CLI/replay surface gesture <pan|fling|pinch|rotate>.
  • Removes public top-level aliases for the newly added pan, fling, and rotate-gesture commands; existing pinch compatibility remains internal and undocumented.
  • Keeps provider/progress accounting and user-facing errors aligned with the gesture command group.
  • Adds iOS runner rotate gesture execution, including opposite-direction rotation handling.
  • Routes iOS pan/fling through the existing runner drag command instead of adding a separate runner fling protocol command.
  • Adds the test-app Gesture lab with the React logo image and live x/y/scale/rotation/fling metrics.
  • Adds replay, integration, unit, SkillGym, CLI help, and docs coverage for the gesture command group.
  • Related follow-up for runner overlays and raw recordings: Fix iOS simulator recording overlays so gesture commands can run during recording #574

Validation

Verified earlier on iPhone 17 with Expo Go on Metro port 8082. The recorded proof sequence completed pinch in/out, pan left/right/up/down, rotate 145 degrees, rotate -215 degrees, and fling left/right/up/down. Final app metrics after the recorded run: x 72, y 56, scale 0.60, rotate -69, fling 10.

Current checks:

  • pnpm format
  • pnpm check:quick
  • pnpm check:fallow --base origin/main
  • pnpm test:integration:progress:check
  • pnpm test:smoke (4 passed, 2 loopback-listener skips in this environment)
  • pnpm exec vitest run src/core/__tests__/dispatch-interactions.test.ts src/core/__tests__/dispatch-pinch.test.ts src/utils/__tests__/args.test.ts test/integration/provider-scenarios/ios-lifecycle.test.ts test/integration/provider-scenarios/android-lifecycle.test.ts
  • pnpm exec vitest run src/utils/__tests__/args.test.ts src/replay/__tests__/script.test.ts src/core/__tests__/capabilities.test.ts src/core/__tests__/dispatch-interactions.test.ts test/integration/provider-scenarios/ios-lifecycle.test.ts test/integration/provider-scenarios/android-lifecycle.test.ts

pnpm check:unit was also rerun. In sandbox it hit environment restrictions (listen EPERM, Swift/tooling failures). Escalated, it reduced to one unrelated failure in src/utils/__tests__/video.test.ts where the MP4 fallback assertion returned false.

Proof video: /private/tmp/agent-device-full-gesture-video/full-gesture-sequence.mp4

Touched 40 files; scope stayed within gesture command support, runner execution, test-app coverage, and command/docs guidance.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 21, 2026

PR Preview Action v1.8.1

QR code for preview link

🚀 View preview at
https://callstackincubator.github.io/agent-device/pr-preview/pr-576/

Built to branch gh-pages at 2026-05-21 14:38 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 053d003033

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +1073 to +1077
let target = interactionRoot(app: app)
switch direction {
case "up":
target.swipeUp()
case "down":
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Execute fling using requested coordinates on iOS

This implementation ignores the command’s x/y/x2/y2 inputs and performs a generic swipe*() on the interaction root, so fling <direction> <x> <y> [distance] does not actually start from the requested point on iOS. In screens with multiple gesture regions (for example a carousel inside a scroll view), the fling can target the wrong container or trigger unrelated navigation even though the caller supplied explicit coordinates.

Useful? React with 👍 / 👎.

Comment on lines +1335 to +1339
private func performCoordinateRotateGesture(app: XCUIApplication, degrees: Double, x: Double?, y: Double?, velocity: Double) -> RunnerInteractionOutcome {
#if os(iOS)
let target = app.windows.firstMatch.exists ? app.windows.firstMatch : app
let radians = CGFloat(degrees * .pi / 180.0)
target.rotate(radians, withVelocity: CGFloat(velocity))
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Apply rotate-gesture center coordinates in runner

The rotate-gesture command validates and forwards optional center coordinates, but this code never uses x/y and always rotates the top window element. As a result, callers cannot control the gesture center, so rotations intended for off-center targets can affect the wrong UI region while still reporting success.

Useful? React with 👍 / 👎.

Comment thread src/core/capabilities.ts
Comment on lines +105 to +108
fling: {
apple: { simulator: true, device: true },
android: { emulator: true, device: true, unknown: true },
linux: LINUX_NONE,
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Restrict macOS fling capability to supported platforms

The capability matrix marks fling as supported for all Apple device kinds, which includes macOS sessions, but the runner implementation explicitly returns unsupported on macOS. This causes command admission to report support and then fail later at execution time, producing avoidable runtime errors and inconsistent platform capability behavior.

Useful? React with 👍 / 👎.

@thymikee thymikee force-pushed the codex/gesture-commands-testbed branch from 053d003 to 5ed3c06 Compare May 21, 2026 12:08
@thymikee thymikee force-pushed the codex/gesture-commands-testbed branch from 5ed3c06 to 2d8d8dc Compare May 21, 2026 12:13
@thymikee thymikee force-pushed the codex/gesture-commands-testbed branch from b62ccef to 222f753 Compare May 21, 2026 13:20
@thymikee thymikee force-pushed the codex/gesture-commands-testbed branch from 222f753 to 9b592f6 Compare May 21, 2026 13:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant