feat: Implement Phase 13 - CI/CD Integration#30
Merged
Conversation
- Add pytest plugin for RAG evaluation (openagent_eval/cicd/plugin.py) - Add threshold-based test gating (openagent_eval/cicd/thresholds.py) - Add CI/CD models (openagent_eval/cicd/models.py) - Add 'oaeval test' CLI command with threshold support - Add CI/CD documentation (docs/13_cicd_integration.md) - Add GitHub Actions workflow example (.github/workflows/eval.yml) - Update .ai/ documentation files to reflect Phase 13 completion - Update CLI shell completions for test command
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| from __future__ import annotations | ||
|
|
||
| import time | ||
| from pathlib import Path |
| except Exception as e: | ||
| duration = time.time() - start_time | ||
| # Create a failed result | ||
| evaluator = ThresholdEvaluator(cicd_config) |
- Add tests for CI/CD models (ThresholdConfig, EvaluationGate, CICDConfig, TestResult) - Add tests for threshold evaluation logic (ThresholdEvaluator) - Add tests for CLI test command validation - Add tests for plugin structure - Fix floating point comparison in NEQ test - Coverage: models 100%, thresholds 95%, plugin 31%
|
|
||
| def test_all_operators(self): | ||
| """Test creating thresholds with all operators.""" | ||
| for op in ThresholdOperator: |
| @@ -0,0 +1,79 @@ | |||
| """Unit tests for CLI test command.""" | |||
|
|
|||
| import pytest | |||
Comment on lines
+5
to
+10
| from openagent_eval.cicd.models import ( | ||
| CICDConfig, | ||
| EvaluationGate, | ||
| ThresholdConfig, | ||
| ThresholdOperator, | ||
| ) |
| @@ -0,0 +1,294 @@ | |||
| """Unit tests for threshold evaluation.""" | |||
|
|
|||
| import pytest | |||
added 2 commits
July 11, 2026 18:35
- Strip ANSI codes in CLI help tests for reliable text matching - Fix eval.yml workflow: audit command uses positional arg, not --corpus
This workflow is a documentation example only (shown in docs/13_cicd_integration.md). It requires knowledge_base/ and config.yaml which don't exist in the repo. Keeping it in .github/workflows/ caused CI failures on every push.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements Phase 13: CI/CD Integration for OpenAgent Eval.
Changes
New Components
pytest plugin (openagent_eval/cicd/plugin.py)
Threshold evaluator (openagent_eval/cicd/thresholds.py)
CI/CD models (openagent_eval/cicd/models.py)
oaeval test CLI command (openagent_eval/cli/commands/test.py)
CI/CD documentation (docs/13_cicd_integration.md)
GitHub Actions workflow (.github/workflows/eval.yml)
Usage Examples
`�ash
Run evaluation with a threshold gate
oaeval test config.yaml -t faithfulness:gte:0.8
Multiple thresholds
oaeval test config.yaml
-t faithfulness:gte:0.8
-t answer_relevancy:gte:0.7
-t latency:lte:5000
JSON output for CI/CD parsing
oaeval test config.yaml -t faithfulness:gte:0.8 --json
`
Files Modified
Breaking Changes
None
Testing
Remaining TODOs
Assumptions & Limitations