🐞: [JUnit 5] Incorrect suite status in Allure report when @BeforeAll fails on first run but passes on retry

[yevhenii-odd]

What happened?

We are using JUnit 5 with Gradle and the test-retry-gradle-plugin plugin to rerun failed tests. We encountered an issue with Allure reports where a test class is still marked as "failed" even though all tests passed on retry.

Problem
If @BeforeAll fails in the first run, no tests inside the class are executed.
On retry, @BeforeAll succeeds, and all tests pass.

In the Allure report:

• All individual tests are marked as passed ✅.
• But the entire test class is still marked as failed ❌.

😕 This creates confusion in reports and CI pipelines, because the actual final result is 100% passed.

✅ Desired Behavior
If @BeforeAll fails during the first attempt but succeeds on retry, it should not appear in the Allure report.

If @BeforeAll fails during all retry attempts, the current behavior is correct — the test class should be marked as failed.

The easiest way to reproduce:

1. Create a test class like this:

@TestInstance(TestInstance.Lifecycle.PER_CLASS)
class RetryBeforeAllTest {

    @BeforeAll
    void beforeAll() {
        throw new RuntimeException("Simulated failure in @BeforeAll");
    }

    @Test
    void test1() {
        Assertions.assertTrue(true);
    }

    @Test
    void test2() {
        Assertions.assertTrue(true);
    }

    @Test
    void test3() {
        Assertions.assertTrue(true);
    }

}

2. Run the test suite once — it will fail due to @BeforeAll.
3. Then comment out the exception in @BeforeAll, and run the test suite again.
4. Generate Allure report (execute allureServe for example)

What Allure Integration are you using?

allure-junit5

What version of Allure Integration you are using?

2.29.1

What version of Allure Report you are using?

2.29.1

Code of Conduct

• I agree to follow this project's Code of Conduct