Skip to content

feat: Add DeepEvalHandler for third-party evaluator integration#528

Open
stone-coding wants to merge 10 commits into
aws:mainfrom
stone-coding:deepeval-handler
Open

feat: Add DeepEvalHandler for third-party evaluator integration#528
stone-coding wants to merge 10 commits into
aws:mainfrom
stone-coding:deepeval-handler

Conversation

@stone-coding

@stone-coding stone-coding commented Jun 16, 2026 •
edited
Loading

Copy link
Copy Markdown

Issue P446281164 — Third-Party Evaluator Integration (Phase 1)

Description of changes Generic BaseAdapter framework that adapts 3P evaluation libraries into AgentCore-compatible Lambda handlers. Supports DeepEval, Autoevals, and is extensible for future libraries (RAGAS, etc.).

Key components:
BaseAdapter — shared orchestration: parse event (supports EvaluatorInput from @custom_code_based_evaluator() decorator) → extract fields → validate → execute with timeout → error handling
DeepEvalAdapter — runs any DeepEval BaseMetric. DeepEvalHandler alias for backward compat.
AutoevalsAdapter — runs any Autoevals scorer
Field extraction from _eval_log_records in ADOT spans (input, actual_output, retrieval_context, expected_output)
Thread-based timeout (default 290s)
field_mapper escape hatch for custom span extraction

Design decisions:
Composes with @custom_code_based_evaluator() decorator — accepts EvaluatorInput directly
Never raises unhandled exceptions — always returns valid response dict
Adding a new library = one ~20 line subclass with execute() method

@haomiao037
Add DeepEvalHandler integration with unit tests
ba80889 
Introduces a new integrations/deepeval/ module that adapts AgentCore
Lambda evaluation events into DeepEval LLMTestCase objects, runs any
BaseMetric, and returns structured score/label/explanation responses.
@haomiao037
Fix span extraction to use real AgentCore _eval_log_records structure
b0d9682
@haomiao037
Set context field from tool messages for HallucinationMetric support
81a46dd
@haomiao037
Use metric.success for label instead of manual threshold comparison
3080e40
@haomiao037
Add model override and timeout enforcement to DeepEvalHandler
34674bb
@haomiao037
Add model override, timeout enforcement, use metric.success, fix Sing…
6aedcbf 
…leTurnParams deprecation
@haomiao037
Fix _get_required_params to handle GEval unmappable typing params
2260eb3
@haomiao037
Add .deepeval/ to gitignore
14f0354
@haomiao037
Move model override to init to avoid per-call mutation
b109a64
@stone-coding stone-coding requested a review from a team June 16, 2026 22:46
@jariy17 jariy17 mentioned this pull request Jun 18, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@stone-coding @haomiao037