Skip to content

fix: improve JSON parsing resilience in synthesis module#35

Merged
himanshu231204 merged 1 commit into
mainfrom
fix/synthesis-json-parsing
Jul 11, 2026
Merged

fix: improve JSON parsing resilience in synthesis module#35
himanshu231204 merged 1 commit into
mainfrom
fix/synthesis-json-parsing

Conversation

@himanshu231204

Copy link
Copy Markdown
Contributor

Summary

  • Enhance _parse_response() in QuestionGenerator to handle malformed JSON from LLM
  • Add JSON array extraction from responses with extra text before/after
  • Fix trailing commas before ] or } in LLM JSON output
  • Replace single quotes with double quotes when no double quotes present
  • Apply same fixes to AdversarialTestCaseGenerator
  • Add 10 new tests covering trailing commas, extra text, single quotes, markdown code blocks, and multiple items
  • All 31 tests pass

Changes

  • openagent_eval/synthesis/question_gen.py: Enhanced _parse_response() with regex extraction and JSON fixes
  • openagent_eval/synthesis/adversarial.py: Applied same JSON resilience fixes
  • ests/unit/test_synthesis/test_question_gen.py: Added 5 new tests for edge cases
  • ests/unit/test_synthesis/test_adversarial.py: Added 5 new tests for edge cases
  • examples/corpus_and_related_modules.ipynb: Added demo notebook using real Groq LLM

- Enhance _parse_response() in QuestionGenerator to handle malformed JSON from LLM
- Add JSON array extraction from responses with extra text before/after
- Fix trailing commas before ] or } in LLM JSON output
- Replace single quotes with double quotes when no double quotes present
- Apply same fixes to AdversarialTestCaseGenerator
- Add 10 new tests covering trailing commas, extra text, single quotes, markdown code blocks, and multiple items
- All 31 tests pass
@himanshu231204 himanshu231204 merged commit da9977c into main Jul 11, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant