FEAT Add RegexScorer and CredentialLeakScorer for regex-based secret detection

[francose]

Closes #1703

Adds two new true/false scorers for fast, regex-based content detection — no LLM call required.

RegexScorer (general purpose)

A reusable TrueFalseScorer that evaluates text against a dict of named regex patterns and returns True if any of them match. Patterns are compiled once in __init__. The score rationale lists which named patterns matched, and categories can be set to tag results (e.g. ["pii"], ["security"]). Aggregator defaults to TrueFalseScoreAggregator.OR but is configurable.

This is intended as a building block for any domain-specific regex check — credentials, PII, profanity, internal identifiers, etc. — without re-implementing the scorer plumbing each time.

CredentialLeakScorer (built on RegexScorer)

Subclasses RegexScorer with a built-in default pattern set covering the most common leaked-credential formats:

• AWS Access Key IDs and Secret Access Keys
• GitHub tokens (ghp_ / gho_ / ghu_ / ghs_ / ghr_)
• Google API keys
• Slack tokens and webhook URLs
• JWTs
• Private key headers (RSA / EC / DSA / OpenSSH)
• Azure storage keys
• Connection strings (mongodb, postgres, mysql, redis, amqp)
• Generic api_key= / secret= / password= / token= assignments

Pass a custom patterns dict to override the defaults entirely (useful for organization-specific secret formats like internal API key prefixes). Category defaults to ["security"].

Because there's no LLM call, scoring runs in microseconds per evaluation, which makes it practical for CI and batch evaluation of thousands of responses.

Other changes

• Exports both scorers from pyrit.score
• Adds a Jupytext doc notebook doc/code/scoring/credential_leak_scorer.py walking through detection, clean responses, and custom patterns
• Unit tests for RegexScorer (match / no-match / multiple matches / category propagation) and CredentialLeakScorer (true positives across all default pattern types, true negatives, rationale content, custom patterns, and memory integration)

[Copilot]

Pull request overview

Adds a new deterministic True/False scorer (CredentialLeakScorer) to quickly detect common credential/secret formats in LLM outputs using compiled regexes, plus unit tests and a public export from pyrit.score.

Changes:

• Introduces CredentialLeakScorer with a default regex pattern set and optional custom patterns.
• Adds unit tests covering true positives/negatives, rationale output, custom patterns, and CentralMemory integration.
• Exposes CredentialLeakScorer from pyrit.score.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

File	Description
pyrit/score/true_false/credential_leak_scorer.py	New regex-based scorer implementation producing true/false Score results with rationale.
tests/unit/score/test_credential_leak_scorer.py	Unit tests validating detection behavior, rationale, custom patterns, and memory integration.
pyrit/score/__init__.py	Exports CredentialLeakScorer from the public pyrit.score package.

[francose]

@microsoft-github-policy-service agree

[francose]

@romanlutz Thank you for the feedback 🙏 — totally agree. The regex matching logic is generic enough to stand on its own.

I'll refactor into:

• RegexScorer — base class that takes patterns: dict[str, str], compiles them, scores against matches, returns rationale with the matched pattern name
• CredentialLeakScorer — thin subclass that just passes the default credential patterns to RegexScorer.__init__

That way spinning up new regex-based scorers (PII detection, code patterns, etc.) is just a new subclass with a different pattern set — no engine duplication.

Will push the update.

[francose]

@romanlutz Pushed the refactor! RegexScorer is now the base class and CredentialLeakScorer just passes its default patterns to super. I also added tests for RegexScorer directly and all existing tests still pass. Let me know if this is what you had in mind 🙏

[romanlutz]

Thanks for this contribution! Approving provided the comments are addressed.