test(e2e): add insights and online-insights lifecycle tests#1599
Open
notgitika wants to merge 3 commits into
Open
test(e2e): add insights and online-insights lifecycle tests#1599notgitika wants to merge 3 commits into
notgitika wants to merge 3 commits into
Conversation
Adds end-to-end coverage for the Lens/Insights feature shipped in the NYS summit release. Two independent sequential suites in e2e-tests/insights-lifecycle.test.ts, each owning its own deployed agent and CFN stack so a deploy failure in one suite does not blank the other: - online-insights lifecycle: add online-insights -> deploy -> invoke -> pause -> resume -> teardown. Verifies live executionStatus toggling through the control plane. - run-insights and recommendation chain: deploy -> invoke -> run insights (async) -> view list -> view detail -> archive -> run insights --wait -> run recommendation --from-insights -> teardown. Covers async submission, local job storage, view/archive round-trip, and the chain from a completed insights job into a system-prompt recommendation. The chain step accepts either success or a service error indicating the upstream job had no usable sessions, since real trace volume is not guaranteed seconds after invoke; flag-parsing errors fail hard.
github-actions
Bot
added
the
agentcore-harness-reviewing
agentcore-devx-automation
Bot
added
the
claude-security-reviewing
Contributor
|
Claude Security Review: no high-confidence findings. (run) |
agentcore-devx-automation
Bot
removed
the
claude-security-reviewing
Contributor
Package TarballHow to installgh release download pr-1599-tarball --repo aws/agentcore-cli --pattern "*.tgz" --dir /tmp/pr-tarball
npm install -g /tmp/pr-tarball/aws-agentcore-0.20.2.tgz |
agentcore-cli-automation
suggested changes
Jun 22, 2026
agentcore-cli-automation
left a comment
agentcore-cli-automation
left a comment
There was a problem hiding this comment.
Test additions look thorough and follow the existing evals-lifecycle.test.ts pattern. One blocker before merge: the view insights <id> test asserts a region field that the CLI doesn't actually emit. Otherwise the wiring against the real commands looks correct.
github-actions
Bot
removed
the
agentcore-harness-reviewing
JobRecordBase does not store region — region is parsed from arn. The view insights <id> --json output therefore has no region field. Switch the detail-call assertion to match arn against the bedrock-agentcore ARN shape and update the InsightsJobJson interface accordingly.
agentcore-devx-automation
Bot
added
the
claude-security-reviewing
Contributor
|
Claude Security Review: no high-confidence findings. (run) |
agentcore-devx-automation
Bot
removed
the
claude-security-reviewing
…on step
The BatchEvaluation API rejects `--insights` and `--evaluator` together
("evaluators and insights are mutually exclusive"), so the --wait submit
exited 1 with success:false and waitJobId stayed undefined. The follow-up
recommendation chain test then ran with id=undefined and tripped the
recommendation handler's 'exactly one evaluator' check. Pass --evaluator
on the recommendation invocation instead, where the handler actually
requires it for system-prompt type.
agentcore-devx-automation
Bot
added
the
claude-security-reviewing
Contributor
|
Claude Security Review: no high-confidence findings. (run) |
agentcore-devx-automation
Bot
removed
the
claude-security-reviewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds e2e tests for insights feature that was recently shipped. two independent
describe.sequentialsuites in a single file (e2e-tests/insights-lifecycle.test.ts), each owning its own deployed agent and CFN stack so a deploy failure in one suite does not blank the other.Suite A:
online-insightslifecycleadd online-insights → deploy → invoke → pause → resume → teardown. Mirrorsevals-lifecycle.test.ts. Verifies liveexecutionStatustoggling through the control plane.Suite B:
run insights+ recommendation chaindeploy → invoke → run insights (async) → view list → view detail → archive → run insights --wait → run recommendation --from-insights → teardown. Covers:run insights(no--wait) and local job recordview insightslist + per-id detailarchive insightsround-trip (job no longer appears inview insights)run insights --waitreaching a terminal statusrun recommendation --from-insightschaining off the completed insights jobThe chain step accepts either
success: trueor a service error containingno sessions/completed_with_errors/failed/empty, since real trace volume is not guaranteed seconds after invoke. Flag-parsing errors (e.g.Unknown option,--from-insights) still fail the test hard, so the wiring is verified even when the upstream job has nothing to learn from.