Subject: Re: [PATCH 0/6 + 1] test-lib: make --verbose output valid TAP

From: SZEDER Gábor <[email-masked]>
Date: Tue, 9 Mar 2021 23:07:17 +0100

On Tue, Mar 09, 2021 at 10:03:54PM +0100, Ævar Arnfjörð Bjarmason wrote:
> 
> On Tue, Mar 09 2021, SZEDER Gábor wrote:
> 
> > On Tue, Mar 09, 2021 at 05:02:12PM +0100, Ævar Arnfjörð Bjarmason wrote:
> >> The eventual goal not included in this series is to have multiple
> >> output targets, and e.g. convert ci/print-test-failures.sh to use a
> >> TAP parser.
> >> 
> >> Machine-readable "TAP --verbose -x" output can bring us a lot of nice
> >> things down the line, I have some local WIP code that's a smarter
> >> version of ci/print-test-failures.sh that knows how to spew out only
> >> the output relevant to the failing test(s).
> >
> > I wonder what you mean by the word "relevant" here, as I can't imagine
> > how you could possibly identify what is relevant for a failing test
> > and what isn't.  If you didn't at all meant "relevant", but that it
> > will show only the output of the failing test(s), then this is a bad
> > example.  Our test cases depend too much on previous test cases, and a
> > failure of one test can be the result of a change in a previous
> > successful tests.
> >
> > Therefore, any such change to 'ci/print-test-failures.sh' will have my
> > firm NACK.
> 
> On e.g. the github CI every step in the run is an collapsable button, so
> we could have our cake and eat it too here.

Well, one of the things I didn't like in GitHub CI is that I had to
click a lot to get to the information I wanted...

> It seems to me like a sane default would be to have an equivalent to
> "print-test-failures.sh" that only prints the --verbose output for the
> failing tests, and a "print-all-output-for-failing-tests.sh" or whatever
> which gave you the full output.
> 
> Even then, there seem to me to be some low hanging fruit for
> abbreviating even that output. E.g. if you we have 100 tests and we
> failed only on the 5th, isn't the --verbose -x output up to and
> including the 5th going to be enough, or do we need it for the other 95?

I have a test fix waiting to be sent out, where a failing git command
in the 'test_when_finished' block of test #21 went unnoticed because
of the broken && chain, and ultimately caused the failure of test
#91.

In my opinion your proposed changes to 'ci/print-test-failures.sh'
would make things worse.