Subject: Re: [RFC PATCH v2 0/6] test-tool: add unit test suite runner

From: Junio C Hamano <[email-masked]>
Date: Sat, 03 Feb 2024 10:52:39 -0800

Josh Steadmon <[email-masked]> writes:

> Please note: this series has been rebased onto jk/unit-tests-buildfix.
>
> For various reasons (see discussion at [1]) we would like an alternative
> to `prove` for running test suites (including the unit tests) on
> Windows.
>
> This series extends the existing `test-tool run-command testsuite` to
> support running unit tests. In addition, it includes some small
> cleanups:
> * move t-basic out of the unit-tests directory
> * don't hardcode the shell for running tests in `test-tool ... testsuite`
> * don't hardcode a test name filter in `test-tool ... testsuite`
> * add a test wrapper script to allow unit tests and the shell test suite
>   to run in a single `prove` process
>
> Some known remaining bits of work:
> * We should investigate switching the Windows CI to use `test-tool`
>   instead of prove. However, Windows CI seems broken on
>   jk/unit-tests-buildfix, and I haven't had time to determine why.

Thanks to Dscho who figured this out, the jk/unit-tests-buildfix
topic in my tree has been updated to pass "win test (n)" jobs.

> * We should determine whether it is confusing or otherwise harmful to
>   people's workflow to have the unit tests run in parallel with shell
>   tests when using prove as the default test target.

I do not know much about "confusing" thing, but if the user
allocates, say, 16 jobs to run tests in parallel, and one of them
drives the "unit test suite runner" that wants to do its own
parallelism, we'd easily end up busting the resource limit the
end-user desires.  It does not necessarily mean that we should limit
the parallelism of "unit test suite runner" to 1 under prove, though.