Skip to content

investigate(ci): diagnose non-deterministic self-scan results across Dependabot root bumps #97

Description

@jeremyjs

Problem

The dogfood CI job "Self-scan (CLI from source)" (dogfood.yml) produces different exit codes on PRs that contain only package.json and pnpm-lock.yaml changes with no modifications to scanner logic, rules, or source files.

Observed pattern (all from June 15, 2026 Dependabot wave):

PR Change Self-scan result
#89 turbo bump (root) ✅ Pass
#91 tailwind bump (root) ✅ Pass
#94 next bump (root) ✅ Pass
#96 @types/node bump (root) ✅ Pass
#88 semantic-release bump (root) ❌ Fail
#90 next bump (apps/web) ❌ Fail
#92 tailwind bump (apps/web) ❌ Fail
#93 @types/node bump (apps/web) ❌ Fail
#95 @types/node bump (packages/cli) ❌ Fail

No rule files, no source files, no action.yml changes. The only differences are package.json and pnpm-lock.yaml contents.

Hypotheses

A (most likely): The false-positive patterns tracked in #78, #79, and #80 are present in the repo source, and the scan result depends on whether Semgrep or Trivy fetches an updated rule DB at runtime. If the DB updates between runs, different PRs trip the FP and others don't.

B: The pnpm-lock.yaml content is being scanned by a native rule. Version bumps in the lock file alter the matched content (e.g., package URLs, integrity hashes) and trigger or suppress a finding.

C: Something in the semantic-release, next, tailwind, or @types/node package installation changes what files are present during the build step, and those files get scanned.

Scope

This is an investigation issue — no code changes in this issue itself.

  1. Download and compare workflow run log artifacts from one passing PR (e.g. chore(deps)(deps-dev): bump turbo from 2.9.16 to 2.9.18 in the lint-and-format group #89) and one failing PR (e.g. chore(deps)(deps-dev): bump semantic-release from 25.0.3 to 25.0.5 in the semantic-release group across 1 directory #88) — look for the specific rule ID, file path, and matched string that caused the exit code 1 on the failing run.
  2. Run two local scans on the same SHA (the current main HEAD) back-to-back and confirm whether they produce identical results. If not, the issue is a non-deterministic scanner DB fetch.
  3. Document the root cause.
  4. File or link a follow-up issue with the fix.

Acceptance Criteria

Non-goals

  • Fixing the root cause in this issue
  • Changing CI workflow structure

QA Checklist

  • Identified specific finding(s) that cause exit code 1 on failing PRs
  • Confirmed or falsified: two runs on same SHA → same exit code
  • Root cause documented in issue comment
  • Follow-up issue filed or existing issue linked

Risk

Low — investigation only

Labels

area:ci, area:scanners, reliability:determinism, investigation, risk:low

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions