Subject: Re: test-lib.sh musings: test_expect_failure considered harmful

From: Junio C Hamano <[email-masked]>
Date: Tue, 12 Oct 2021 09:45:19 -0700

Ævar Arnfjörð Bjarmason <[email-masked]> writes:

> On Mon, Oct 11 2021, Junio C Hamano wrote:
>
> [Removed "In-reply-to: <[email-masked]>" with the Subject
> change]

Please do not do the former, although it is welcome to change Subject.

> Presumably with test_expect_failure.
>
> I'll change it, in this case we'd end up with a test_expect_success at
> the end, so it doesn't matter much & I don't care.

I do agree with you that compared to expect_success, which requires
_all_ steps to succeed, so an failure in any of its steps is
immediately noticeable, it is harder to write and keep
expect_failure useful, because it is not like we are happy to see
any failure in any step.  We do not expect a failure in many
preparation and conclusion steps in the &&-chain in expect_failure
block, and we consider it is an error if these steps fail.  We only
want to mark only a single step to exhibit an expected but undesirable
behaviour.

But even with the shortcomings of expect_failure, it still is much
better than claiming that we expect a bogus outcome.

Improving the shortcomings of expect_failure would be a much better
use of our time than advocating an abuse of expect_sucess, I would
think.