Subject: Re: [PATCH v2 34/35] tests: start asserting that *.txt SYNOPSIS matches -h output

From: Junio C Hamano <[email-masked]>
Date: Wed, 28 Sep 2022 13:27:33 -0700

Ævar Arnfjörð Bjarmason  <[email-masked]> writes:

> There's been a lot of incremental effort to make the SYNOPSIS output
> in our documentation consistent with the -h output,
> e.g. cbe485298bf (git reflog [expire|delete]: make -h output
> consistent with SYNOPSIS, 2022-03-17) is one recent example, but that
> effort has been an uphill battle due to the lack of regression
> testing.
>
> This adds such regression testing, we can parse out the SYNOPSIS
> output with "sed", and is turns out it's relatively easy to normalize
> it and the "-h" output to match on another.
>
> We now ensure that we won't have regressions when it comes to the list
> of commands in "expect_help_to_match_txt" below, and in subsequent
> commits we'll make more of them consistent.

Call that file as such, not a plain "list".  It might be useful to
have it in Documentation/ or somewhere outside the test script so
that people who work on making the match know where to look, as you
are updating CodingGuidelines for this change anyway.

> --- a/Documentation/CodingGuidelines
> +++ b/Documentation/CodingGuidelines
> @@ -634,7 +634,8 @@ Writing Documentation:
>  
>   A few commented examples follow to provide reference when writing or
>   modifying command usage strings and synopsis sections in the manual
> - pages:
> + pages. The two should match, see t/t0450-txt-doc-vs-help.sh for
> + (partial) regression testing.

> +test_expect_success 'setup: list of txt v.s. help is sorted' '
> +	sort -u list >list.sorted &&
> +	if ! test_cmp list list.sorted
> +	then
> +		BUG "please keep the command list sorted"
> +	fi
> +'

If the list becomes an external file, we probably could add a
"recommended pre-commit hook" for developers to reduce mistakes,
protecting us even from developers who forgets to run tests.

Or even a "clean" filter that automatically sorts, specified via the
attribute for that file, but that wouldn't protect us from careless
developers who are unlikely to enable the filter X-<.

> +while read builtin
> +do
> +...
> +done <builtins

Fun!