Skip to content

Collect coverage only on the primary CI job; use the sys.monitoring coverage core#264

Merged
cboos merged 2 commits into
mainfrom
dev/ci-windows-coverage
Jul 4, 2026
Merged

Collect coverage only on the primary CI job; use the sys.monitoring coverage core#264
cboos merged 2 commits into
mainfrom
dev/ci-windows-coverage

Conversation

@cboos

@cboos cboos commented Jul 4, 2026

Copy link
Copy Markdown
Collaborator

Part of #248.

Problem

All 10 CI matrix jobs ran pytest with --cov (xml + html + term, across the unit/TUI/browser step chain), but only the primary job (ubuntu 3.12) uploads the reports and feeds the PR coverage comment — the other 9 computed coverage and discarded it. Measured cost of those flags on the CI unit-test selection: +67% pytest wall time under the default C tracer (worst on Windows, where the unit step ran 734s vs 161s on ubuntu, run 28703179425).

Changes

  • ci.yml: coverage flags move to job-level COV/COV_APPEND env vars that resolve to empty on non-primary jobs, preserving the --cov-append chain (unit → TUI → browser → benchmark) on the primary job. Workflow validated with actionlint.
  • pyproject.toml: core = "sysmon" in [tool.coverage.run] — the sys.monitoring core (Py 3.12+) cuts the remaining coverage tax on the primary job from +67% to +14%, with byte-identical coverage totals (84.71%). On Py < 3.12 coverage warns once and falls back to the default core; moot in CI since the only coverage job is on 3.12. Side benefit: de-flakes test_render_performance_under_threshold, which can exceed its threshold under the C tracer. COVERAGE_CORE=ctrace still overrides for A/B timing.
  • Coverage floor bumped to 7.9 (introduced the [run] core setting; older versions warn and silently keep the slow core). uv.lock already resolves 7.12.0.

Does dropping Windows coverage lose anything?

No. Diffing covered lines (full local Windows run vs the primary job's coverage artifact at the same commit) shows Windows uniquely covers exactly 5 lines, all in cli.py: the SIGUSR1 no-op return and the drive-letter branch of convert_project_path_to_claude_dir. Those were already misses in the published report, which has always been ubuntu-only — so reported coverage changes by exactly 0 lines. (If cross-OS merged coverage is ever wanted, the groundwork is noted in the ci.yml comment; relative_files = true is already set.)

Expected effect

  • Non-primary jobs (all 5 Windows ones included): ~65% pytest wall-time cut on the unit step, plus removal of the ~40–50s coverage-combine/report tail after each of the three pytest steps.
  • Primary job: coverage tax drops from +67% to +14%.

Verification on this PR's CI run

  • A non-primary job's pytest invocation shows no --cov flags in the log
  • The primary job still uploads both coverage artifacts and posts the PR coverage comment
  • Windows unit step wall time drops from ~10–12 min toward ~4–5 min

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Chores
    • Updated the CI workflow to centralize test coverage settings and apply them only on the primary test job.
    • Adjusted coverage tooling settings to use a newer coverage engine and aligned the minimum supported coverage package version.

cboos and others added 2 commits July 4, 2026 17:38
All 10 matrix jobs ran pytest with --cov, but only the primary job
(ubuntu 3.12) uploads the reports and feeds the PR comment — the other
9 computed coverage and discarded it, at ~65% extra pytest wall time
(worst on Windows, where the unit step took 734s vs 161s on ubuntu).

Coverage flags move to job-level COV/COV_APPEND env vars that resolve
to empty on non-primary jobs, preserving the --cov-append chain across
the unit/TUI/browser/benchmark steps on the primary job.

Measured impact on reported coverage: none. Diffing covered lines
(local Windows full run vs the primary artifact of run 28703179425,
same commit) shows Windows uniquely covers only 5 lines — the SIGUSR1
no-op return and the drive-letter branch of
convert_project_path_to_claude_dir in cli.py — and those were already
misses in the published report, which has always been ubuntu-only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ivalent)

Set core = "sysmon" in [tool.coverage.run] rather than a CI env var:
one source of truth that also speeds up local `just test-cov` and
de-flakes test_render_performance_under_threshold, which can exceed its
threshold under the C tracer's overhead. Measured on the CI unit-test
selection: +67% wall time under the C tracer vs +14% under sysmon.

On Py < 3.12 coverage emits a single cosmetic no-sysmon warning and
falls back to the default core — moot in CI, where the only coverage
job (primary, ubuntu 3.12) is on 3.12+. Bump the coverage floor to 7.9,
which introduced the [run] core setting; older versions warn
'unrecognized option' and silently fall back to the slow default core,
losing the speedup. uv.lock already resolved 7.12.0, so only non-uv
installs are affected by the floor.

COVERAGE_CORE env var still overrides the setting for A/B timing
(e.g. COVERAGE_CORE=ctrace).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 4, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The CI workflow now derives pytest coverage flags from job-level environment variables (COV, COV_APPEND) enabled only on the primary matrix job, replacing hardcoded flags. pyproject.toml sets Coverage.py's sysmon core and raises the coverage dependency minimum version to 7.9.

Changes

CI Coverage Configuration

Layer / File(s) Summary
CI env-based coverage flags
.github/workflows/ci.yml
Adds job env variables COV/COV_APPEND gated on matrix.is-primary and replaces hardcoded --cov/--cov-append arguments in unit, TUI, browser, and benchmark pytest steps with these env references.
Coverage.py sysmon core and version bump
pyproject.toml
Sets [tool.coverage.run] core = "sysmon" with explanatory comments and raises the coverage[toml] dev dependency requirement from >=7.6.0 to >=7.9.

Estimated code review effort: 1 (Trivial) | ~5 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the two main changes: primary-job-only coverage collection and switching to the sys.monitoring coverage core.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch dev/ci-windows-coverage

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
.github/workflows/ci.yml (1)

53-69: 🔒 Security & Privacy | 🔵 Trivial | 💤 Low value

Silence zizmor template-injection warnings by using shell env vars instead of ${{ env.* }}.

Since COV/COV_APPEND are already exported as job-level env: vars, the run: steps can reference them as shell variables ($COV, $COV_APPEND) instead of GitHub Actions expression syntax. This is the documented zizmor remediation for template-injection: expression expansion happens before the shell sees the script, whereas shell variable substitution goes through normal quoting rules. Although the actual risk here is low (values are derived only from matrix.is-primary, not attacker input), this removes all 4 flagged findings with a trivial change.

🔧 Proposed fix
     - name: Run unit tests (coverage on primary only)
       # -p no:playwright: unit tests don't use the browser fixtures; skipping the
       # plugin avoids a ~1s playwright import per xdist worker, which Windows pays
       # per spawned worker. See work/xdist-import-cost.md.
-      run: uv run pytest -p no:playwright -m "not (tui or browser or benchmark)" ${{ env.COV }}
+      run: uv run pytest -p no:playwright -m "not (tui or browser or benchmark)" $COV

     - name: Run TUI tests (coverage append on primary only)
-      run: uv run pytest -m tui ${{ env.COV_APPEND }}
+      run: uv run pytest -m tui $COV_APPEND

     - name: Run browser tests (coverage append on primary only)
-      run: uv run pytest -m browser ${{ env.COV_APPEND }}
+      run: uv run pytest -m browser $COV_APPEND

     - name: Run benchmark tests with coverage append (primary only)
       if: matrix.is-primary
       env:
         CLAUDE_CODE_LOG_DEBUG_TIMING: "1"
-      run: uv run pytest -m benchmark ${{ env.COV_APPEND }} -v
+      run: uv run pytest -m benchmark $COV_APPEND -v
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/ci.yml around lines 53 - 69, The pytest workflow steps
still interpolate COV and COV_APPEND with GitHub Actions expressions, which
triggers zizmor template-injection warnings; update the Run unit tests, Run TUI
tests, Run browser tests, and Run benchmark tests steps to reference the
already-exported shell environment variables instead. In the ci workflow, keep
the job-level env definitions for COV/COV_APPEND and change the run commands to
use shell expansion in those step scripts so the values are resolved by the
shell rather than by `${{ env.* }}`.

Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In @.github/workflows/ci.yml:
- Around line 53-69: The pytest workflow steps still interpolate COV and
COV_APPEND with GitHub Actions expressions, which triggers zizmor
template-injection warnings; update the Run unit tests, Run TUI tests, Run
browser tests, and Run benchmark tests steps to reference the already-exported
shell environment variables instead. In the ci workflow, keep the job-level env
definitions for COV/COV_APPEND and change the run commands to use shell
expansion in those step scripts so the values are resolved by the shell rather
than by `${{ env.* }}`.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 920941a7-7574-4ca1-8812-d3f86d745b17

📥 Commits

Reviewing files that changed from the base of the PR and between fe2b746 and 295fb56.

⛔ Files ignored due to path filters (1)
  • uv.lock is excluded by !**/*.lock
📒 Files selected for processing (2)
  • .github/workflows/ci.yml
  • pyproject.toml

@cboos

cboos commented Jul 4, 2026

Copy link
Copy Markdown
Collaborator Author

Windows 3.12 and its 21m... Fable called that a "lemon VM" and losing at the Windows runner lottery :P

But hey, the Linux numbers did drop again, so I guess we're good with these changes (initially aimed at reducing the gap between Windows and Linux build times!)

@cboos

cboos commented Jul 4, 2026

Copy link
Copy Markdown
Collaborator Author

(Claude) Re the zizmor template-injection nit on the ${{ env.COV }} interpolations: declining, deliberately.

  • No injection surface: COV/COV_APPEND are workflow-literal ternaries over matrix.is-primary — a boolean defined by the matrix itself. No attacker-controllable context (github.event.*, inputs, etc.) can reach these values, which is the threat the zizmor rule targets.
  • The suggested $COV form has a cross-shell trap in this matrix: default run: shell is bash on ubuntu but pwsh on windows-latest, where $COV is an undefined PowerShell variable (env vars are $env:COV). It would only work today by coincidence (all Windows jobs happen to want the empty value); if a Windows job ever runs coverage (e.g. a future cross-OS coverage merge), the flags would silently vanish. ${{ env.* }} expands before either shell parses the script, so it behaves identically across the whole matrix — that shell-agnosticism is why it was chosen.

Making shell expansion safe would need shell: bash on all four steps — more CI change than the trivial-severity finding warrants.

@cboos cboos merged commit 52da5c9 into main Jul 4, 2026
17 checks passed
@cboos cboos deleted the dev/ci-windows-coverage branch July 4, 2026 17:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant