From: Junio C Hamano <gitster@pobox•com>
To: Jeff King <peff@peff•net>
Cc: git@vger•kernel.org
Subject: Re: [RFC] test-lib: detect common misuse of test_expect_failure
Date: Mon, 17 Oct 2016 10:36:19 -0700 [thread overview]
Message-ID: <xmqqa8e2x4h8.fsf@gitster.mtv.corp.google.com> (raw)
In-Reply-To: <20161014235718.2rbejrzs5axhopt7@sigill.intra.peff.net> (Jeff King's message of "Fri, 14 Oct 2016 19:57:18 -0400")
Jeff King <peff@peff•net> writes:
> I like the general idea, but I'm not sure how this would interact with
> the tests in t0000 that test the test suite.
I tried but gave up adding a new test for this to t0000 ;-)
>> test_expect_failure () {
>> + if test "$test_in_progress" = 1
>> + then
>> + error "bug in the test script: did you mean test_must_fail instead of test_expect_failure?"
>> + fi
>
> This follows existing practice for things like the &&-lint-checker, and
> bails out on the whole test script.
Yes, you guessed correctly where the above came from.
> That sometimes makes it hard to find
> the problematic test, especially if you're running via something like
> "prove", because it doesn't make valid TAP output.
Yeah, true.
> It might be nicer if we just said "this test is malformed, and therefore
> fails", and then you get all the usual niceties for recording and
> finding the failed test.
>
> I don't think it would be robust enough to try to propagate the error up
> to the outer test_expect_success block (and anyway, you'd also want to
> know about it in a test_expect_failure block; it's a bug in the test,
> not a known breakage). But perhaps error() could dump some TAP-like
> output with a "virtual" failed test.
>
> Something like:
> ...
> which lets "make prove" collect the broken test number.
>
> It would perhaps need to cover the case when $test_count is "0"
> separately. I dunno. It would be nicer still if we could continue
> running other tests in the script, but I think it's impossible to
> robustly jump back to the outer script.
>
> These kinds of "bug in the test suite" are presumably rare enough that
> the niceties don't matter that much, but I trigger the &&-checker
> reasonably frequently (that and test_line_count, because I can never
> remember the correct invocation).
>
> Anyway. That's all orthogonal to your patch. I just wondered if we could
> do better, but AFAICT the right way to do better is to hook into
> error(), which means your patch would not have to care exactly how it
> fails.
Yeah, the change to error() may be a good thing to do, but it has
quite a many callers in t/*lib*.sh and definitely deserves to be a
separate patch, not tied to this single test.
prev parent reply other threads:[~2016-10-17 17:36 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-10-14 22:38 [RFC] test-lib: detect common misuse of test_expect_failure Junio C Hamano
2016-10-14 23:57 ` Jeff King
2016-10-17 17:36 ` Junio C Hamano [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=xmqqa8e2x4h8.fsf@gitster.mtv.corp.google.com \
--to=gitster@pobox$(echo .)com \
--cc=git@vger$(echo .)kernel.org \
--cc=peff@peff$(echo .)net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox