feat: Phase 12 — Synthetic Test Data Generator#29
Merged
Conversation
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| Combined list of all adversarial test case types. | ||
| """ | ||
| all_cases: list[TestCase] = [] | ||
| for test_type in TestCaseType: |
| async def test_generate_all_types(self) -> None: | ||
| """Test generating all adversarial types.""" | ||
| responses = {} | ||
| for test_type in TestCaseType: |
|
|
||
| from openagent_eval.exceptions.synthesis import SynthesisExecutionError | ||
| from openagent_eval.synthesis.generator import SyntheticDataGenerator, _chunk_text, _read_corpus | ||
| from openagent_eval.synthesis.models import TestCase, TestCaseType |
| TestCase, | ||
| TestCaseType, | ||
| ) | ||
| from openagent_eval.synthesis.generator import _chunk_text |
Add synthesis module for generating synthetic Q&A test cases from documents: - QuestionGenerator: LLM-based question generation from text chunks - AdversarialTestCaseGenerator: 5 adversarial types (unanswerable, ambiguous, misleading, multi-hop, counterfactual) - SyntheticDataGenerator: orchestrator with concurrent LLM calls - oaeval synth CLI command with Rich output and file save (json/jsonl/huggingface) - 56 unit and integration tests
himanshu231204
force-pushed
the
feature/synthetic-test-data
branch
from
e87e2ec to
1f54200
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add openagent_eval/synthesis/ module for generating synthetic Q&A test cases from documents. This enables developers to bootstrap evaluation datasets from their knowledge base without manual test case creation.
What's Included
New Module: openagent_eval/synthesis/
CLI Command
Exceptions
Tests
Usage
`�ash
Generate from a text file
oaeval synth --input corpus.txt --num-questions 20 --output dataset.json
Generate with adversarial cases
oaeval synth --input corpus.txt --adversarial --adversarial-types unanswerable,misleading
Generate from a directory of documents
oaeval synth --input ./docs/ --num-questions 50 --format jsonl
`
Architecture
Follows existing patterns from diagnosis/ module:
Checklist