feat: add 11 new evaluators, expand standard suites, fix evaluator validator gap#198
Open
jithin23-kv wants to merge 4 commits into
Open
Conversation
…ator, reconcile severities
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
WalkthroughThe PR expands evaluator coverage, revises severity and standards metadata, adds derived EU AI Act, NIST, and OWASP API suites, supports multiple evaluator YAML layouts during validation, and synchronizes generated catalogs and harmful-content suite membership. ChangesEvaluator catalog expansion
Estimated code review effort: 5 (Critical) | ~120 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
jithin23-kv
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
scripts/validate-skills.ts (1)
43-55: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
scripts/validate-skills.ts: add the newstandardskeys toEvaluatorYamlSchema
standardsstill only allowsowasp-llm,owasp-agentic, andatlas, but the evaluator YAMLs now useeu-ai-act,nist,owasp-api, andowasp-mcpin many places. Zod strips unknown object keys by default, so those values are silently dropped instead of validated. Extend the schema to cover the new metadata.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/validate-skills.ts` around lines 43 - 55, Extend the standards object in EvaluatorYamlSchema to accept the eu-ai-act, nist, owasp-api, and owasp-mcp keys as optional strings alongside the existing standards. Ensure these metadata fields are validated and preserved instead of stripped.Source: Coding guidelines
runners/extension/catalog.json (1)
4-219: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winRegenerate
runners/extension/catalog.jsonto includeowasp-api-top10andnist-ai-rmf
The currentsuitesarray still only hasowasp-agentic-ai,owasp-llm-top10, andowasp-mcp-top10; the loader already derivesowasp-api-top10fromstandards.owasp-apiandnist-ai-rmffromstandards.nist, so this checked-in catalog is missing those derived suites.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@runners/extension/catalog.json` around lines 4 - 219, Regenerate the catalog’s suites array so it includes the derived owasp-api-top10 and nist-ai-rmf suites alongside the existing OWASP suites. Preserve the loader-generated definitions and evaluator mappings from standards.owasp-api and standards.nist rather than manually inventing or altering unrelated suite entries.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@evaluators/agent/bias/bias-race/evaluator.yaml`:
- Around line 24-25: Update the standards object in EvaluatorYamlSchema within
validate-skills.ts to explicitly accept the nist and eu-ai-act string fields, or
apply a string-valued catchall if standards keys are intentionally dynamic, so
parsing preserves these evaluator metadata entries.
In `@evaluators/agent/harmful/harmful-copyright-violations/evaluator.yaml`:
- Around line 1-28: Add the required standards metadata to the
harmful-copyright-violations evaluator alongside id, name, and severity, using
the established standards structure and applicable standard identifier(s) from
comparable evaluator definitions so it is included in derived standard suites.
In `@evaluators/agent/harmful/harmful-hate/harmful-hate.test.yaml`:
- Around line 7-8: Update the fail_case fixture to use a concrete, unambiguous
hateful response targeting a protected group instead of the “[demeaning
stereotypes follow]” placeholder, while preserving the expected FAIL outcome.
In `@evaluators/agent/harmful/harmful-illegal-activities/evaluator.yaml`:
- Around line 5-6: Update the standards definition in EvaluatorYamlSchema within
validate-skills.ts to explicitly accept the nist key, or configure an
appropriate catchall, so parsed evaluator YAML preserves nist and the catalog
loader includes the evaluator in the nist suite.
In `@skills/mcp-redteaming/opfor-setup/catalog.json`:
- Around line 665-674: Correct the swapped OWASP API mappings: update the ssrf
entry in skills/mcp-redteaming/opfor-setup/catalog.json (665-674) and
runners/extension/catalog.json (813-818) to use API7, and update the
debug-access entry in skills/agent-redteaming/opfor-setup/catalog.json (561-570)
to use API8. Also apply the corresponding mapping changes in
evaluators/mcp/injection/ssrf/evaluator.yaml and
evaluators/agent/disclosure/debug-access/evaluator.yaml.
---
Outside diff comments:
In `@runners/extension/catalog.json`:
- Around line 4-219: Regenerate the catalog’s suites array so it includes the
derived owasp-api-top10 and nist-ai-rmf suites alongside the existing OWASP
suites. Preserve the loader-generated definitions and evaluator mappings from
standards.owasp-api and standards.nist rather than manually inventing or
altering unrelated suite entries.
In `@scripts/validate-skills.ts`:
- Around line 43-55: Extend the standards object in EvaluatorYamlSchema to
accept the eu-ai-act, nist, owasp-api, and owasp-mcp keys as optional strings
alongside the existing standards. Ensure these metadata fields are validated and
preserved instead of stripped.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 9b4bff47-fa76-4872-bc8c-3739d64215cb
📒 Files selected for processing (146)
core/src/catalog/loadEvaluatorCatalog.tsevaluators/agent/access-control/bfla/evaluator.yamlevaluators/agent/access-control/rbac/evaluator.yamlevaluators/agent/accuracy/hallucination/evaluator.yamlevaluators/agent/accuracy/misinformation/evaluator.yamlevaluators/agent/accuracy/overreliance/evaluator.yamlevaluators/agent/bias/bias-age/evaluator.yamlevaluators/agent/bias/bias-disability/evaluator.yamlevaluators/agent/bias/bias-gender/evaluator.yamlevaluators/agent/bias/bias-political/bias-political.test.yamlevaluators/agent/bias/bias-political/evaluator.yamlevaluators/agent/bias/bias-political/patterns/abortion-ban-loaded-framing.yamlevaluators/agent/bias/bias-political/patterns/conservative-persuasion.yamlevaluators/agent/bias/bias-political/patterns/gun-control-loaded-framing.yamlevaluators/agent/bias/bias-political/patterns/objectively-better-party.yamlevaluators/agent/bias/bias-political/patterns/partisan-essay-left.yamlevaluators/agent/bias/bias-political/patterns/partisan-essay-right.yamlevaluators/agent/bias/bias-political/patterns/progressive-persuasion.yamlevaluators/agent/bias/bias-political/patterns/voting-recommendation.yamlevaluators/agent/bias/bias-race/evaluator.yamlevaluators/agent/bias/bias-religious/bias-religious.test.yamlevaluators/agent/bias/bias-religious/evaluator.yamlevaluators/agent/bias/bias-religious/patterns/comparative-violence-claim.yamlevaluators/agent/bias/bias-religious/patterns/policy-favoritism.yamlevaluators/agent/bias/bias-religious/patterns/practice-disparagement.yamlevaluators/agent/bias/bias-religious/patterns/religion-ranking.yamlevaluators/agent/bias/bias-religious/patterns/trustworthiness-generalization.yamlevaluators/agent/brand-conduct/competitors/evaluator.yamlevaluators/agent/brand-conduct/contracts/evaluator.yamlevaluators/agent/brand-conduct/imitation/evaluator.yamlevaluators/agent/brand-conduct/off-topic/evaluator.yamlevaluators/agent/code-execution/improper-output-handling/evaluator.yamlevaluators/agent/code-execution/shell-injection/evaluator.yamlevaluators/agent/code-execution/sql-injection/evaluator.yamlevaluators/agent/disclosure/debug-access/evaluator.yamlevaluators/agent/disclosure/pii-api-db/evaluator.yamlevaluators/agent/disclosure/pii-direct/evaluator.yamlevaluators/agent/disclosure/pii-session/evaluator.yamlevaluators/agent/disclosure/pii-social/evaluator.yamlevaluators/agent/disclosure/pii-social/patterns/authority-plus-urgency-stack.yamlevaluators/agent/disclosure/pii-social/patterns/fabricated-emergency.yamlevaluators/agent/disclosure/pii-social/patterns/impersonated-it-support.yamlevaluators/agent/disclosure/pii-social/patterns/impersonated-law-enforcement.yamlevaluators/agent/disclosure/pii-social/patterns/impersonated-manager.yamlevaluators/agent/disclosure/pii-social/patterns/self-identity-confusion.yamlevaluators/agent/disclosure/pii-social/patterns/trust-building-then-ask.yamlevaluators/agent/disclosure/pii-social/pii-social.test.yamlevaluators/agent/disclosure/sensitive-disclosure/evaluator.yamlevaluators/agent/disclosure/system-prompt-leakage/evaluator.yamlevaluators/agent/excessive-agency/excessive-agency/evaluator.yamlevaluators/agent/excessive-agency/tool-misuse/evaluator.yamlevaluators/agent/harmful/harmful-bioweapons/evaluator.yamlevaluators/agent/harmful/harmful-child-exploitation/evaluator.yamlevaluators/agent/harmful/harmful-copyright-violations/evaluator.yamlevaluators/agent/harmful/harmful-copyright-violations/patterns/book-excerpt-reproduction.yamlevaluators/agent/harmful/harmful-copyright-violations/patterns/chunked-incremental-reproduction.yamlevaluators/agent/harmful/harmful-copyright-violations/patterns/movie-script-reproduction.yamlevaluators/agent/harmful/harmful-copyright-violations/patterns/paywalled-article-reproduction.yamlevaluators/agent/harmful/harmful-copyright-violations/patterns/proprietary-source-code.yamlevaluators/agent/harmful/harmful-copyright-violations/patterns/song-lyrics-reproduction.yamlevaluators/agent/harmful/harmful-copyright-violations/patterns/translation-evasion.yamlevaluators/agent/harmful/harmful-cybercrime-malicious-code/evaluator.yamlevaluators/agent/harmful/harmful-harassment-bullying/evaluator.yamlevaluators/agent/harmful/harmful-harassment-bullying/patterns/anonymous-humiliation.yamlevaluators/agent/harmful/harmful-harassment-bullying/patterns/group-chat-mockery.yamlevaluators/agent/harmful/harmful-harassment-bullying/patterns/insult-escalation.yamlevaluators/agent/harmful/harmful-harassment-bullying/patterns/prank-framing.yamlevaluators/agent/harmful/harmful-harassment-bullying/patterns/roleplay-as-the-bully.yamlevaluators/agent/harmful/harmful-harassment-bullying/patterns/third-party-proxy-request.yamlevaluators/agent/harmful/harmful-hate/evaluator.yamlevaluators/agent/harmful/harmful-hate/harmful-hate.test.yamlevaluators/agent/harmful/harmful-hate/patterns/academic-framing.yamlevaluators/agent/harmful/harmful-hate/patterns/crescendo-escalation-multi-turn.yamlevaluators/agent/harmful/harmful-hate/patterns/debate-prep-framing.yamlevaluators/agent/harmful/harmful-hate/patterns/direct-request-baseline.yamlevaluators/agent/harmful/harmful-hate/patterns/fictional-character-wrapper.yamlevaluators/agent/harmful/harmful-hate/patterns/neutral-discussion-counter-probe.yamlevaluators/agent/harmful/harmful-hate/patterns/satire-wrapper.yamlevaluators/agent/harmful/harmful-illegal-activities/evaluator.yamlevaluators/agent/harmful/harmful-illegal-activities/patterns/elder-targeted-scam-script.yamlevaluators/agent/harmful/harmful-illegal-activities/patterns/fake-charity-scheme.yamlevaluators/agent/harmful/harmful-illegal-activities/patterns/fake-invoice.yamlevaluators/agent/harmful/harmful-illegal-activities/patterns/fraud-prevention-inversion.yamlevaluators/agent/harmful/harmful-illegal-activities/patterns/identity-fraud.yamlevaluators/agent/harmful/harmful-illegal-activities/patterns/incremental-scam-assembly.yamlevaluators/agent/harmful/harmful-illegal-activities/patterns/tech-support-scam-script.yamlevaluators/agent/harmful/harmful-illegal-drugs/evaluator.yamlevaluators/agent/harmful/harmful-indiscriminate-weapons/evaluator.yamlevaluators/agent/harmful/harmful-misinformation-disinformation/evaluator.yamlevaluators/agent/harmful/harmful-misinformation-disinformation/patterns/conspiracy-narrative.yamlevaluators/agent/harmful/harmful-misinformation-disinformation/patterns/election-disinformation.yamlevaluators/agent/harmful/harmful-misinformation-disinformation/patterns/fabricated-health-scare.yamlevaluators/agent/harmful/harmful-misinformation-disinformation/patterns/fabricated-news-article.yamlevaluators/agent/harmful/harmful-misinformation-disinformation/patterns/fabricated-quote.yamlevaluators/agent/harmful/harmful-misinformation-disinformation/patterns/fabricated-statistic.yamlevaluators/agent/harmful/harmful-misinformation-disinformation/patterns/fictional-framing-evasion.yamlevaluators/agent/harmful/harmful-non-violent-crime/evaluator.yamlevaluators/agent/harmful/harmful-non-violent-crime/patterns/counterfeit-manufacturing-fictional.yamlevaluators/agent/harmful/harmful-non-violent-crime/patterns/counterfeit-manufacturing.yamlevaluators/agent/harmful/harmful-non-violent-crime/patterns/money-laundering.yamlevaluators/agent/harmful/harmful-non-violent-crime/patterns/prescription-forgery.yamlevaluators/agent/harmful/harmful-non-violent-crime/patterns/retail-theft-evasion.yamlevaluators/agent/harmful/harmful-privacy/evaluator.yamlevaluators/agent/harmful/harmful-privacy/patterns/address-lookup-guidance.yamlevaluators/agent/harmful/harmful-privacy/patterns/covert-location-tracking.yamlevaluators/agent/harmful/harmful-privacy/patterns/covert-recording.yamlevaluators/agent/harmful/harmful-privacy/patterns/dossier-compilation.yamlevaluators/agent/harmful/harmful-privacy/patterns/incremental-dossier-aggregation.yamlevaluators/agent/harmful/harmful-privacy/patterns/medical-history-exposure.yamlevaluators/agent/harmful/harmful-privacy/patterns/professional-pretext-cover.yamlevaluators/agent/harmful/harmful-privacy/patterns/stalkerware-parental-pretext.yamlevaluators/agent/harmful/harmful-radicalization/evaluator.yamlevaluators/agent/harmful/harmful-self-harm/evaluator.yamlevaluators/agent/harmful/harmful-sex-crime/evaluator.yamlevaluators/agent/harmful/harmful-sexual-content/evaluator.yamlevaluators/agent/harmful/harmful-sexual-content/patterns/age-ambiguity-boundary-probe.yamlevaluators/agent/harmful/harmful-sexual-content/patterns/continue-the-story-escalation.yamlevaluators/agent/harmful/harmful-sexual-content/patterns/direct-request.yamlevaluators/agent/harmful/harmful-sexual-content/patterns/euphemism-escalation.yamlevaluators/agent/harmful/harmful-sexual-content/patterns/fictional-novel-framing.yamlevaluators/agent/harmful/harmful-sexual-content/patterns/roleplay-wrapper.yamlevaluators/agent/harmful/harmful-specialized-advice/evaluator.yamlevaluators/agent/harmful/harmful-unsafe-practices/evaluator.yamlevaluators/agent/harmful/harmful-violent-crime/evaluator.yamlevaluators/agent/harmful/harmful-weapons-ied/evaluator.yamlevaluators/agent/injection/agent-goal-hijack/evaluator.yamlevaluators/agent/injection/ascii-smuggling/evaluator.yamlevaluators/agent/injection/hijacking/evaluator.yamlevaluators/agent/injection/jailbreaking/evaluator.yamlevaluators/agent/injection/prompt-injection/evaluator.yamlevaluators/agent/mcp-usage/mcp-cross-resource-leakage/evaluator.yamlevaluators/agent/mcp-usage/mcp-intent-subversion/evaluator.yamlevaluators/agent/mcp-usage/mcp-missing-authentication/evaluator.yamlevaluators/agent/mcp-usage/mcp-tool-description-injection/evaluator.yamlevaluators/agent/multi-agent/cascading-failures/evaluator.yamlevaluators/agent/multi-agent/human-agent-trust/evaluator.yamlevaluators/agent/multi-agent/inter-agent-communication/evaluator.yamlevaluators/agent/multi-agent/rogue-agents.yamlevaluators/agent/resource/reasoning-dos/evaluator.yamlevaluators/agent/resource/unbounded-consumption/evaluator.yamlevaluators/mcp/injection/ssrf/evaluator.yamlrunners/extension/catalog.jsonscripts/validate-skills.tsskills/agent-redteaming/opfor-setup/catalog.jsonskills/mcp-redteaming/opfor-setup/catalog.jsonsuites/agent/harmful-content.yaml
💤 Files with no reviewable changes (15)
- evaluators/agent/access-control/rbac/evaluator.yaml
- evaluators/agent/mcp-usage/mcp-missing-authentication/evaluator.yaml
- evaluators/agent/mcp-usage/mcp-cross-resource-leakage/evaluator.yaml
- evaluators/agent/mcp-usage/mcp-tool-description-injection/evaluator.yaml
- evaluators/agent/harmful/harmful-radicalization/evaluator.yaml
- evaluators/agent/multi-agent/human-agent-trust/evaluator.yaml
- evaluators/agent/harmful/harmful-weapons-ied/evaluator.yaml
- evaluators/agent/multi-agent/inter-agent-communication/evaluator.yaml
- evaluators/agent/mcp-usage/mcp-intent-subversion/evaluator.yaml
- evaluators/agent/brand-conduct/contracts/evaluator.yaml
- evaluators/agent/multi-agent/rogue-agents.yaml
- evaluators/agent/harmful/harmful-sex-crime/evaluator.yaml
- evaluators/agent/multi-agent/cascading-failures/evaluator.yaml
- evaluators/agent/access-control/bfla/evaluator.yaml
- evaluators/agent/injection/hijacking/evaluator.yaml
achuvyas-kv
approved these changes
Jul 14, 2026
There was a problem hiding this comment.
🧹 Nitpick comments (1)
runners/extension/scripts/build-catalog.mjs (1)
292-297: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftSuite derivation correctly mirrors
loadEvaluatorCatalog.ts.Verified against
core/src/catalog/loadEvaluatorCatalog.ts— the newowasp-api/nistgroups, theowasp-api-top10block, theeu-ai-actrename, and thenist-ai-rmfblock match the TS loader's contract exactly.One structural concern worth noting: this function is now a hand-maintained duplicate of
deriveStandardSuitesincore/src/catalog/loadEvaluatorCatalog.ts(JS vs TS). The two must be kept in lockstep manually — the very rename this PR performs (eu-ai-act-bias→eu-ai-act) had to be applied in both places, and the past review's API7/API8 swap fix likewise spans multiple generated files. Consider generating this script's suite-derivation table from a single shared source (e.g. a small JSON/YAML config both the TS loader and this build script read), or having this script shell out to/import the compiled TS logic, to eliminate the drift risk going forward.Also applies to: 321-331, 364-383
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@runners/extension/scripts/build-catalog.mjs` around lines 292 - 297, Eliminate the duplicated suite-derivation definitions in the build script and core loader by introducing a single shared source of truth, such as shared JSON/YAML configuration consumed by both `deriveStandardSuites` implementations. Update `build-catalog.mjs` and `loadEvaluatorCatalog.ts` to derive their suite tables from that source while preserving the current groups and mappings, including `owasp-api`, `eu-ai-act`, `nist`, `owasp-api-top10`, and `nist-ai-rmf`.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@runners/extension/scripts/build-catalog.mjs`:
- Around line 292-297: Eliminate the duplicated suite-derivation definitions in
the build script and core loader by introducing a single shared source of truth,
such as shared JSON/YAML configuration consumed by both `deriveStandardSuites`
implementations. Update `build-catalog.mjs` and `loadEvaluatorCatalog.ts` to
derive their suite tables from that source while preserving the current groups
and mappings, including `owasp-api`, `eu-ai-act`, `nist`, `owasp-api-top10`, and
`nist-ai-rmf`.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 100cd752-177c-4e47-9592-20105b8d80a1
📒 Files selected for processing (7)
evaluators/agent/harmful/harmful-hate/harmful-hate.test.yamlevaluators/mcp/injection/ssrf/evaluator.yamlrunners/extension/catalog.jsonrunners/extension/scripts/build-catalog.mjsscripts/validate-skills.tsskills/agent-redteaming/opfor-setup/catalog.jsonskills/mcp-redteaming/opfor-setup/catalog.json
🚧 Files skipped from review as they are similar to previous changes (4)
- evaluators/mcp/injection/ssrf/evaluator.yaml
- evaluators/agent/harmful/harmful-hate/harmful-hate.test.yaml
- scripts/validate-skills.ts
- skills/agent-redteaming/opfor-setup/catalog.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The evaluator catalog was missing coverage for several real-world harm categories (hate speech, harassment, sexual content, fraud/scams, disinformation, privacy violations, non-violent crime, copyright, social-engineering PII disclosure, political/religious bias), and the standard-suite derivation (
owasp-api,nist,eu-ai-act) was incomplete. Separately,scripts/validate-skills.tswas silently skipping an entire class of evaluator files, so some evaluators were never actually validated.Solution
harmful/,disclosure/, andbias/, each with hand-authored pass/fail criteria and attack patterns, reviewed independently per-evaluator for rigor and consistency with house style.owasp-apiandnist, and broadenedeu-ai-actbeyond bias-only coverage.suites/agent/harmful-content.yamlto include the new harmful evaluators.scripts/validate-skills.ts, which only discovered directory-form (evaluator.yaml) evaluators and silently skipped flat-file-form ones (inlinepatterns) — it now covers both, matching the runtime loader's discovery logic.standards:frontmatter and severity values across evaluators for internal consistency.Changes
core/src/catalog/loadEvaluatorCatalog.ts— addowasp-api/nistderived suites, broadeneu-ai-actscripts/validate-skills.ts— discover + validate flat-file evaluators (inlinepatterns)evaluators/agent/harmful/{harmful-hate,harmful-harassment-bullying,harmful-sexual-content,harmful-illegal-activities,harmful-misinformation-disinformation,harmful-privacy,harmful-non-violent-crime,harmful-copyright-violations}/— new evaluatorsevaluators/agent/disclosure/pii-social/,evaluators/agent/bias/{bias-political,bias-religious}/— new evaluatorssuites/agent/harmful-content.yaml— add the new harmful evaluatorsevaluator.yamlfiles —standards:tag fixes and severity consistencyskills/*/opfor-setup/catalog.json,runners/extension/catalog.json— regeneratedIssue
N/A
How to test
npm run validate:skills # 108 files, 0 errors (8 pre-existing warnings on patternless source-analysis evaluators) npm run build:catalog npm run extension:catalog npx tsc -b core npm run lint npm run format:checkSummary by CodeRabbit
New Features
Improvements
Validation
Documentation