NoSuchElementExceptions mark test as broken

[farious2009]

Describe the Bug

Hi there,

Maybe it is intended to be as this and not a bug, but wanna re-ask why this unchecked exception marks tests as being broken?

I have expected the test to fail and being rerun as usual, because the code is not a part of preconfig, but rather - realization already:

Steps to Reproduce

Just to run a simple method with a predefined exception.

Expected Behaviour

Tests should fail

Screenshots or Additional Context

No response

What Language are you using?

Java

What Framework/Allure Integration you are using?

Allure TestNG

What version of Allure Integration you are using?

2.27.0

What version of Allure Report you are using?

2.27.0

Code of Conduct

• I agree to follow this project's Code of Conduct

[baev]

Allure uses Status.FAILED for errors that extend java.lang.AssertionError. All the other errors are considered as Status.BROKEN.

From Allure Report documentation:

A failed test (red) is a test that encountered an unexpected behavior in the system under test. This means that the test itself seems to be valid (not Broken), but its execution ended with a false assertion.

A broken test (yellow) is a test that failed because of a test defect. Unlike Failed, this status means that the test was unable to check the product's behavior as it intended to, therefore, the failure may or may not indicate an actual product defect.

From the test design perspective, in your case:

If you check that certain actions should produce availableDays with at least one element, you should use assert instead of throwing a NoSuchElementException:

assertThat(availableDays).hasSize(1);

The other case is when it is usually non-empty, and the collection itself is not the part for the tested functionality (e.g. you test that appointment reservation works, and you prepared the data with some available days), you may assume availableDays isn't empty and throw NoSuchElementExceptions otherwise. In that scenario, the broken status shows you that the desired functionality wasn't tested, and functionality may work or may not, so additional attention (test retry) is required.

[farious2009]

Thanks.

In theory I do not need to assert all part of a product while running a single test, because we depend on many upstream services, and it may end up in redundant assertions; I thought, if I had added manually at least a custom message this would indicate this is me, so I expect it could fail and being rerun.

Also, it sounds like only AssertionError matters here and other exceptions won't be needed ,as anyway it will lead to status 'broken' what will make rerun being impossible (why would I need to use any others exceptions at all?..).

Can I use at least custom exceptions what will extend java.lang.AssertionError to make test fail?

[baev]

Can I use at least custom exceptions what will extend java.lang.AssertionError to make test fail?

Yes, you can

[farious2009]

Ideally, I would like to have a control on what could be considered as expected error and behaves as AssertionError.
Let's say I have a list of exceptions which marks tests as broken, and the rest (off it) - as broken

__
But, if at least, I could extend AssertionError with custom exceptions it would be a workaround either.

Thanks.