test: flag e2e_ha_full as flake under HA pipelining#23541
Merged
Conversation
http://ci.aztec-labs.com/136431da99834194 dequeued PR #23344 again with the HA full suite failing under proposer pipelining. The visible failure signals in the dashboard log are recurring "Timed out waiting for block with archive matching checkpoint proposal" warnings across slots and an "Error building checkpoint at slot 127: already proposed block for slot 127 index 0" on HA-4, so multiple peers race on the same proposal under pipelining. The existing per-assertion / afterAll-hook flake entries for this suite did not match because jest's per-test banner is not reached within the dashboard log capture. Add a broad regex-only entry to flag any failure of yarn-project/end-to-end/scripts/run_test.sh ha src/composed/ha/e2e_ha_full.test.ts as a flake until the HA pipelining work stabilises.
PhilWindle
approved these changes
May 24, 2026
This was referenced May 24, 2026
PhilWindle
pushed a commit
that referenced
this pull request
May 24, 2026
## Summary PR #23344 was dequeued again at https://github.com/AztecProtocol/aztec-packages/actions/runs/26364977301. The flake-flag added in #23541 wasn't enough — ci3 retries once and the HA full suite fails both attempts. Multiple distinct failure modes have been hit recently: - `✕ should coordinate governance voting across HA nodes` - `✕ should distribute work across multiple HA nodes` - afterAll teardown hangs → `Exceeded timeout of [0-9]+ ms for a hook` - 5-peer races on checkpoint proposals (`Timed out waiting for block with archive matching checkpoint proposal`, `already proposed block for slot N index 0`) This change collapses the four overlapping `e2e_ha_full` entries in `.test_patterns.yml` into a single `skip: true` entry so the suite is outright skipped on `merge-train/spartan` until HA pipelining stabilises. ## Notes - Targets `merge-train/spartan` per yarn-project default base. - Adjacent `src/e2e_slashing/attested_invalid_proposal.test.ts` entry left untouched. - Owner kept as @spyros (current owner of the suite-level entry). Requested in Slack by @PhilWindle. --- *Created by [claudebox](https://claudebox.work/v2/sessions/a745321353cb58ff) · group: `slackbot`*
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dequeued from merge-train/spartan again: http://ci.aztec-labs.com/136431da99834194.
The HA full suite keeps failing under proposer pipelining with shifting symptoms. In this run the dashboard log shows recurring
validator:proposal-handler Timed out waiting for block with archive matching checkpoint proposalwarnings (slot 98, 115, …) and anError building checkpoint at slot 127: already proposed block for slot 127 index 0on HA-4 — i.e. the 5 HA peers race on the same proposal. The bundled #23539 (parallel peer teardown) and #23524 (afterAll hook timeout) entries did not catch this run because jest's per-test summary was not reached within the dashboard log capture.This PR adds a broad regex-only entry under
.test_patterns.ymlto flag any failure ofyarn-project/end-to-end/scripts/run_test.sh ha src/composed/ha/e2e_ha_full.test.tsas a flake. Owner: @PaLLa, matching the existing pipelining-flavoured entries for this suite.The intent is to unblock the merge queue while the HA pipelining stabilisation work continues; narrow the regex (or add a real fix) once the failure modes settle down.
Created by claudebox · group:
slackbot