Subject: Re: [PATCH 2/2] diff tests: rewrite flakyness-causing test "aid"

From: SZEDER Gábor <[email-masked]>
Date: Thu, 15 Apr 2021 23:26:00 +0200

On Tue, Apr 13, 2021 at 04:55:23PM -0300, Matheus Tavares Bernardino wrote:
> On Tue, Apr 13, 2021 at 4:01 PM Ævar Arnfjörð Bjarmason
> > >> diff --git a/t/t4013-diff-various.sh b/t/t4013-diff-various.sh
> > >> index 67f6411aff9..228ff100c61 100755
> > >> --- a/t/t4013-diff-various.sh
> > >> +++ b/t/t4013-diff-various.sh
> > >> @@ -200,10 +200,12 @@ do
> > >>         esac
> > >>         test=$(echo "$label" | sed -e 's|[/ ][/ ]*|_|g')
> > >>         pfx=$(printf "%04d" $test_count)
> > >> -       expect="$TEST_DIRECTORY/t4013/diff.$test"
> > >> +       expect_relative="t4013/diff.$test"
> > >> +       expect="$TEST_DIRECTORY/$expect_relative"
> > >>         actual="$pfx-diff.$test"
> > >>
> > >>         test_expect_$status "git $cmd # magic is ${magic:-(not used)}" '
> > >> +               test_when_finished "rm $actual" &&
> > >
> > > Nit: before these two patches, "$actual" was only removed when the
> > > test succeeded. So, in case of failure, the failed output files would
> > > still be there for debugging. It might be interesting to keep this
> > > behavior and only remove "$actual" at the end of the test.
> >
> > Either I'm missing something or you are, that's how test_when_finished
> > works.
> >
> > It's skipped under e.g. "--immediate --debug". See b586744a864 (test:
> > skip clean-up when running under --immediate mode, 2011-06-27)
> 
> I was mostly thinking about the `artifacts` zip we get from our CI
> when a test fails. I find the final trash dir quite useful for some
> post-mortem analysis, especially to debug WIP tests that only fail
> occasionally or test failures on OSes I don't have quick access to.

On Travis CI we run tests with '--immediate' for exactly this reason;
I don't know why it's done differently on other CI systems, and,
unfortunately, 'git log --grep=immediate ci/' didn't turn up any
insights.