Subject: Re: [PATCH v3 29/29] CI: make it easy to use ci/*.sh outside of CI

From: Ævar Arnfjörð Bjarmason <[email-masked]>
Date: Thu, 14 Apr 2022 15:51:30 +0200

On Thu, Apr 14 2022, Eric Sunshine wrote:

> On 4/13/22 3:51 PM, Ævar Arnfjörð Bjarmason wrote:
>> In preceding commits the ci/.sh scripts have lost most of their
>> CI-specific assumptions. Let's go even further and explicitly support
>> running ci/lib.sh outside of CI.
>>
>> This was possible before by faking up enough CI-specific variables,
>> but as shown in the new "help" output being added here using the
>> ci/lib.sh to provide "CI-like" has now become trivial.
>>
>> The ci/print-test-failures.sh scripts can now be used outside of CI as
>> well, the only GitHub CI-specific part is now guarded by a check that
>> we'll pass if outside of GitHub CI.
>>
>> There's also a special-case here to not clobber $MAKEFLAGS in the
>> environment if we're outside of CI, in case the user has e.g. "jN" or
>> other flags to "make" that they'd prefer configured already.
>>
>> Signed-off-by: Ævar Arnfjörð Bjarmason <[email-masked]>
>> ---
>> diff --git a/ci/lib.sh b/ci/lib.sh
>> @@ -1,6 +1,30 @@
>> +#  Usage
>> +CI_TYPE_HELP_COMMANDS='
>> +	# run "make all test" like the "linux-leaks" job
>> +	(eval $(jobname=linux-leaks ci/lib.sh --all) && make test)
>> +
>> +	# run "make all test" like the "linux-musl" job
>> +	(eval $(jobname=linux-musl ci/lib.sh --all) && make test)
>> +
>> +	# run "make test" like the "linux-TEST-vars" job (uses various
>   GIT_TEST_* modes)
>> +	make && (eval $(jobname=linux-TEST-vars ci/lib.sh --test) && make test)
>> +
>> +	# run "make test" like the "linux-sha256" job
>> +	make && (eval $(jobname=linux-sha256 ci/lib.sh --test) && make test)
>> +'
>> +
>> +CI_TYPE_HELP="
>> +running $0 outside of CI? You can use ci/lib.sh to set up your
>> +environment like a given CI job. E.g.:
>> +$CI_TYPE_HELP_COMMANDS
>> +
>> +note that some of these (e.g. the linux-musl one) may not work as
>> +expected due to the CI job configuring a platform that may not match
>> +yours."
>> +
>> @@ -9,6 +33,10 @@ mode=$1
>>   if test -z "$mode"
>>   then
>>   	echo "need a $0 mode, e.g. --build or --test" >&2
>> +	if test -z "$CI_TYPE"
>> +	then
>> +		echo "$CI_TYPE_HELP" >&2
>> +	fi
>>   	exit 1
>>   fi
>
> It would never occur to me to try running a script named ci/lib.sh in
> the first place, and I'm not sure I would even think to poke around in
> the `ci` directory with the idea of being able to run CI tasks
> locally. Would it make sense to aid discovery by mentioning this new
> feature in the "GitHub CI" section of SubmittingPatches as a follow up
> to this series? (If so, perhaps the "GitHub CI" section should be
> renamed to "CI".)

Yes, I'd like to do that, but as a follow-up, specifically to have
failing tests in CI suggest a way to run them locally.

It would be better as ci/config.sh or something, but it was ci/lib.sh
already, and this way minimized the diff/conflicts.

>> @@ -76,10 +108,29 @@ CC_PACKAGE=
>>   # How many jobs to run in parallel?
>>   NPROC=10
>> +case "$CI_TYPE" in
>> +'')
>> +	if command -v nproc >/dev/null
>
> You need to redirect stderr too in order to avoid a scary error
> message on platforms lacking `nproc`:
>
>     if command -v nproc >/dev/null 2>&1

I figured the scary error would be useful, it's only if you run it
ad-hoc, so it would be a nice hint as to why we fallback on the default
nproc.

>> +	then
>> +		NPROC=$(nproc)
>> +	else
>> +		NPROC=1
>> +	fi
>
> Neither macOS nor BSD has an `nproc` command, however, they do have
> `sysctl`:
>
>     elif command -v sysctl >/dev/null 2>&1
>     then # macOS & BSD
>         NPROC=$(sysctl -n hw.ncpu 2>/dev/null)
>
> and Windows provides a suitable environment variable:
>
>     elif test -n "$NUMBER_OF_PROCESSORS"
>     then # Windows
>         NPROC="$NUMBER_OF_PROCESSORS"

*nod*

I'd rather not finish this part up though and instead just have this
call some moral equivalent of t/helper/test-online-cpus.c.

Most of our CI is linux, so just nproc for now felt like a good initial
addition which we could expand on later.

>> +	if test -n "$MAKEFLAGS"
>> +	then
>> +		COMMON_MAKEFLAGS="$MAKEFLAGS"
>> +	else
>> +		COMMON_MAKEFLAGS=--jobs=$NPROC
>> +	fi
>> +	;;