Subject: Re: [PATCH 1/1] Makefile: add prove and coverage-prove targets

From: SZEDER Gábor <[email-masked]>
Date: Tue, 29 Jan 2019 22:58:27 +0100

On Tue, Jan 29, 2019 at 03:49:58PM -0500, Derrick Stolee wrote:
> On 1/29/2019 1:10 PM, Derrick Stolee wrote:
> > On 1/29/2019 12:34 PM, SZEDER Gábor wrote:
> >> On Tue, Jan 29, 2019 at 04:58:27PM +0100, SZEDER Gábor wrote:
> >> And in the related email discussion [1]:
> >>
> >>   But even though the docs claim it [-j<N>] should be possible,
> >>   I've been getting "random" test failures when compiled with coverage
> >>   support, that went away with -j1.  So the tests still run with -j1, as
> >>   with the first version of the series.
> >>
> >> So it doesn't seem to be that bad after all, because it's not
> >> "completely breaks" but "random test failures".  Still far from ideal,
> >> but the original coverage patch is just about 3 weeks short of a
> >> decade old, so maybe things have improved since then, and it'd be
> >> worth a try to leave GIT_PROVE_OPTS as is and see what happens.
> > 
> > It would certainly be nice if the build time could be reduced through
> > parallel test runs. I've kicked off a build using GIT_PROVE_OPTS="-j12"
> > to see what happens.
> 
> I did get a failed test with this run:
> 
> t0025-crlf-renormalize.sh                 (Wstat: 256 Tests: 3 Failed: 1)
>   Failed test:  2
>   Non-zero exit status: 1
> 
> This was on the 'jch' branch, and an equivalent build with sequential
> execution did not have this failure. That's flaky enough for me to stick
> to sequential runs.

That failure is not coverage-related, but as it turned out 9e5da3d055
(add: use separate ADD_CACHE_RENORMALIZE flag, 2019-01-17) made t0025
rather flaky:

  https://public-inbox.org/git/[email-masked]/

When reading those old commit messages and discussions in the
afternoon, I was wondering what "random test failures" actually meant,
since it was not stated explicitly that it was coverage-related.  For
all we know it could have been "general" test flakiness that happened
to manifest under the higher load of a parallel test run.