Skip to content

test: fix flaky test_runs_concurrently by asserting peak concurrency instead of wall time#2073

Open
vdusek wants to merge 1 commit into
masterfrom
test/fix-flaky-test-runs-concurrently
Open

test: fix flaky test_runs_concurrently by asserting peak concurrency instead of wall time#2073
vdusek wants to merge 1 commit into
masterfrom
test/fix-flaky-test-runs-concurrently

Conversation

@vdusek

@vdusek vdusek commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

test_runs_concurrently verifies that AutoscaledPool runs tasks concurrently rather than sequentially. It inferred this from the total wall time (elapsed.wall < 0.3), which is a fragile proxy: on a slow CI runner (seen on windows-latest / Python 3.14) event-loop scheduling overhead pushed the wall time to 0.314 s even though all 10 tasks did overlap, tripping the tight 0.3 s threshold.

This replaces the timing proxy with a direct, deterministic assertion of the property the test actually protects: the peak number of tasks running simultaneously must reach the configured concurrency (10). The check no longer depends on wall-clock time, so it can't flake under CI load or on slow runners, and it's stronger than before (a sequential or only partially concurrent pool would peak below 10, whereas the old wall-time check could pass at partial concurrency).

Verified locally: a harness replicating the exact pool setup reached peak 10 in 200/200 iterations under 2x CPU oversubscription; the test passes 0 failures across repeated runs; lint and type checks pass.

@vdusek vdusek added t-tooling Issues with this label are in the ownership of the tooling team. adhoc Ad-hoc unplanned task added during the sprint. labels Jul 14, 2026
@vdusek vdusek self-assigned this Jul 14, 2026
@codecov

codecov Bot commented Jul 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.37%. Comparing base (272bb28) to head (be6c133).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2073      +/-   ##
==========================================
- Coverage   93.41%   93.37%   -0.05%     
==========================================
  Files         179      179              
  Lines       12496    12496              
==========================================
- Hits        11673    11668       -5     
- Misses        823      828       +5     
Flag Coverage Δ
unit 93.37% <ø> (-0.05%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@vdusek vdusek marked this pull request as ready for review July 14, 2026 14:29
@vdusek vdusek requested a review from Pijukatel July 14, 2026 14:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants