feat(scanner): enable SkillSpector LLM semantic pass (Anthropic Claude)

[DevelopmentCats]

What this does

Flips the scheduled scan from --no-llm to SkillSpector's two-stage analyser (static rules + LLM semantic pass).

Provider is anthropic against api.anthropic.com directly, model claude-sonnet-4-6 (SkillSpector's bundled meta_analyzer slot default; 1M context window; SkillSpector's model_registry.yaml already knows its token limits so no runtime fallback warning). SkillSpector cannot be routed through Coder's AI Gateway today, so the Anthropic API key is on a separate billing line from Coder usage — the four-bullet reason chain is in docs/CALIBRATION.md under "Provider choice and the workflow gap." Short version: SkillSpector pipes every provider through langchain_openai.ChatOpenAI (OpenAI /v1/chat/completions shape), aibridge only exposes that path under /openai, and SkillSpector's anthropic provider hardcodes https://api.anthropic.com/v1/ and ignores ANTHROPIC_BASE_URL. Empirically verified end-to-end against the workspace's aibridge.

Stack

Stacked on PR #1. Merge that first, then re-target this to main and merge.

Measured impact on the five in-tree skills

Measured by running skillspector scan twice on each upstream skill (once with --no-llm, once with LLM mode on, model gpt-4.1-mini through aibridge during development). Methodology and the model-swap caveat are in docs/CALIBRATION.md.

Skill	static score	static verdict	LLM score	LLM verdict	findings Δ
coder/coder-modules	10	clean	0	clean	1 → 0 (-1)
coder/coder-templates	10	clean	0	clean	1 → 0 (-1)
coder/modules	0	clean	0	clean	0 → 0
coder/setup	100	malicious	26	clean	23 → 2 (-21)
coder/templates	0	clean	0	clean	0 → 0
TOTAL					25 → 2 (-23)

coder/setup flips malicious → clean. The LLM filtered all 23 static-only findings as context-aware false positives (EA2 hits on safeguard prose, MP2 hits on PNG assets, SC2 hits on curl coder.com/install.sh, PE3 hits on the skill's own scratch files, etc.) and surfaced two new MEDIUM SQP-2 findings: the GitHub device-flow scripts write the OAuth token and session config to disk without an explicit notification. Those are real and minor; the cleanest fix is a one-line echo upstream in coder/skills, not a change here.

Model swap caveat: production runs against Claude Sonnet 4.6, not gpt-4.1-mini. The 25 → 2 delta above measures SkillSpector's LLM semantic pass as a capability; per-finding counts may shift one or two either way under Claude. The verdict-band outcomes are robust to that drift because every static finding on the four other in-tree skills is well below the suspicious_risk_score: 51 cutoff to begin with, so even a 100% no-filter LLM still leaves them clean. Recalibration against Claude lands in a follow-up PR once the secret is wired in and the first production scan runs.

Setup before merge

1. Add ANTHROPIC_API_KEY as a repo secret (from console.anthropic.com).
2. Apply the workflow diff to .github/workflows/scan.yaml (see collapsible below). The Coder Agents app on this repo doesn't have workflows: write, so the bot couldn't commit that file. Either grant the permission and ping me to re-push, or paste the diff yourself.
3. After merge: Actions → scan → Run workflow.

Graceful fallback

If the secret isn't set, the workflow emits a warning and runs --no-llm. A fresh fork keeps producing valid (lower-precision) scans without any setup.

Why scan.yaml isn't in this PR

GitHub Apps need an explicit workflows: write permission separate from contents: write to modify .github/workflows/*. The Coder Agents installation on this repo currently has contents: write but not workflows: write, so the agent could not commit the scan.yaml change. Two ways forward:

• Quick path: paste the diff below into scan.yaml via the GitHub web editor. Takes about a minute.
• Permanent path: grant the Coder Agents GitHub App workflows: write for this repo. After that I can re-push and this PR's workflow file will be included.

Either produces the same merged state.

Workflow file diff (to paste into .github/workflows/scan.yaml)

Three edits inside the scan job.

1. Add new step after Verify skill path exists:

      - name: Determine LLM mode
        id: llm
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          set -euo pipefail
          if [[ -n "${ANTHROPIC_API_KEY:-}" ]]; then
            echo "extra_flags=" >> "$GITHUB_OUTPUT"
            echo "SkillSpector LLM mode: enabled (anthropic provider, api.anthropic.com)." >&2
          else
            echo "extra_flags=--no-llm" >> "$GITHUB_OUTPUT"
            echo "::warning::ANTHROPIC_API_KEY secret not set; SkillSpector will run with --no-llm. See README \"One-time setup\" to enable the LLM semantic pass."
          fi

2. Replace the SkillSpector (JSON) step:

      - name: SkillSpector (JSON)
        if: steps.path_check.outputs.drift == 'false'
        continue-on-error: true
        env:
          SKILLSPECTOR_PROVIDER: anthropic
          SKILLSPECTOR_MODEL: claude-sonnet-4-6
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          mkdir -p out
          skillspector scan "source/${{ matrix.skill_path }}" \
            ${{ steps.llm.outputs.extra_flags }} \
            --format json \
            --output "out/skillspector.json" || true

3. Same swap for SkillSpector (SARIF):

      - name: SkillSpector (SARIF)
        if: steps.path_check.outputs.drift == 'false'
        continue-on-error: true
        env:
          SKILLSPECTOR_PROVIDER: anthropic
          SKILLSPECTOR_MODEL: claude-sonnet-4-6
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          mkdir -p out
          skillspector scan "source/${{ matrix.skill_path }}" \
            ${{ steps.llm.outputs.extra_flags }} \
            --format sarif \
            --output "out/skillspector.sarif" || true

Decision log

• Anthropic API direct over aibridge. Aibridge cannot route SkillSpector at Claude today: SkillSpector's anthropic provider hardcodes api.anthropic.com and ignores ANTHROPIC_BASE_URL; SkillSpector pipes every provider through langchain_openai.ChatOpenAI (OpenAI /v1/chat/completions shape); aibridge does proxy Claude, but only in Anthropic's native /v1/messages shape under its /anthropic path, and does not mount /v1/chat/completions on that path. Verified empirically: POST $ANTHROPIC_BASE_URL/v1/messages returns Claude output; POST $ANTHROPIC_BASE_URL/v1/chat/completions returns route not supported. The trade-off (separate billing line vs Claude-class model) was made deliberately; using openai against aibridge with gpt-4.1-mini is a viable alternative documented in docs/CALIBRATION.md. The provider line flips back to openai without any workflow change if aibridge later adds the missing route or SkillSpector gains a native-Anthropic integration.
• claude-sonnet-4-6. SkillSpector's bundled meta_analyzer slot default; the provider docstring describes that slot as "cheaper for the high-volume filter pass," which is precisely the per-finding intent classification this scanner does. Bumped from the earlier claude-sonnet-4-5-20250929 pick because 4-6 is one generation newer, widens context from 200K to 1M (reduces chunking on larger skills), and is in SkillSpector's model_registry.yaml (no runtime fallback warning). Bare alias is used because the dated suffixes for the 4-6 series return 404 from Anthropic's API as of 2026-06-23; the bare alias is the canonical ID. Override in config.yaml to claude-opus-4-6 (SkillSpector's DEFAULT_MODEL, ~5x cost) or to claude-opus-4-8 / claude-fable-5 if a higher tier is warranted later.
• Calibration table was measured on gpt-4.1-mini. Captured during the aibridge-routing exploration, before the provider swap. Kept in docs/CALIBRATION.md because the LLM-pass capability — not the model choice — is what drives the 25 → 2 reduction, and the verdict-band outcomes are robust to model differences for these five inputs. A follow-up PR refreshes the numbers with Claude.
• Graceful --no-llm fallback. A fresh fork should produce something useful immediately, not 404 the publish pipeline because the operator hasn't added the secret yet. The workflow warning makes the degraded state visible.
• No CI change. ci.yaml uses inline pytest fixtures and never invokes skillspector live, so no inference cost on PR review.
• No schema or verdict-math change. LLM mode shifts which findings reach the verdict, not how the verdict is computed. CALIBRATION.md walks through this explicitly.
• Permissions manifest dropped from the near-term plan. Was originally proposed to fix coder/setup's false-positive avalanche. LLM mode handles that better (23 → 2 findings). The two remaining findings on coder/setup are real and can be fixed in a 4-line PR against coder/skills rather than justified via a manifest. The manifest design is captured for future use when we onboard third-party skills.

This PR was prepared with help from Coder Agents.

[Copilot]

Pull request overview

Updates the scanner configuration and documentation to support running NVIDIA SkillSpector with an optional LLM semantic pass (with a fallback to static-only mode), as part of the scheduled scan pipeline described in this repo.

Changes:

• Extend config.yaml with a scanners.skillspector.llm configuration block and make scanners.skillspector.flags empty by default.
• Document the LLM semantic pass behavior, expected precision delta, and repo one-time setup steps in README.md and docs/CALIBRATION.md.
• Shift responsibility for driving --no-llm to the scheduled workflow (though the workflow change itself is not included in this PR’s diff).

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.

File	Description
README.md	Updates architecture and adds one-time setup guidance, including LLM credential setup.
docs/CALIBRATION.md	Adds an “LLM semantic pass” section explaining expected effects and limitations.
config.yaml	Adds scanners.skillspector.llm block and removes the default --no-llm flag from config-driven flags.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.