Subject: Re: my complaints with clar

From: Phillip Wood <[email-masked]>
Date: Mon, 1 Dec 2025 14:16:13 +0000

Hi Peff

On 30/11/2025 13:46, Jeff King wrote:
> 
>    cl_invoke(check_int_full("0", 0));
>    cl_invoke(check_int_full("11", 11));
>    cl_invoke(check_int_full("-23", -23));
>    cl_invoke(check_int_full("+23", 23));
> 
> gives you the line number in the caller. Better, but there's a lot of
> cross-referencing the line numbers (plus sprinkling cl_invoke everywhere
> is ugly).

The README for the old unit-testing framework recommended wrapping 
helper functions in a macro to pass the file and line numbers from the 
calling site. Perhaps we should do the same with clar

#define check_int_full(input, expect) cl_invoke(check_int_full(input, 
expect))

> What I really would have liked is some notion of "context". If the
> helper could have done:
> 
>    cl_context("input: %.*s", (int)len, buf);
> 
> or similar, and failed assertions print that context, then that would
> have made the failing part of the test easy to see, even without using
> cl_invoke() at all.

If you're writing a helper function you might want to use cl_failf() 
instead of cl_assert_* to provide more context but it's a pain that you 
can't just use the builtin assertions. I've not used them but there are 
assertions named cl_assert_*_ which I think let you add some context. 
One of the features of the conversion of our unit tests from the old 
framework to clar has been a degradation of the diagnostic messages when 
a test fails.

>> +void test_parse_int__basic(void)
>> +{
>> +	cl_invoke(check_int_full("0", 0));
>> +	cl_invoke(check_int_full("11", 11));
>> +	cl_invoke(check_int_full("-23", -23));
>> +	cl_invoke(check_int_full("+23", 23));
>> +	cl_invoke(check_int_str("  31337  ", 7, 0, 31337));
>> +
>> +	cl_invoke(check_int_err("  garbage", EINVAL));
>> +	cl_invoke(check_int_err("", EINVAL));
>> +	cl_invoke(check_int_err("-", EINVAL));
>> +
>> +	cl_invoke(check_int("123", 2, 2, 0, 12));
>> +}
> 
> into a series of nine separate tests, each of which gets a name. But
> each of those tests is at least five lines of boilerplate, which sucks
> (plus you have to come up with syntactically valid C names for them).

Yes that's a pain. One of the nice things about the old framework was 
the the TEST() macro just took an expression and created a test case out 
of it which worked well for tests like this and meant you could have 
table driven tests where each entry in the table was a separate test case.

>> +	/*
>> +	 * Do not use cl_assert_equal_i_fmt(..., PRIuMAX) here. The macro
>> +	 * casts to int under the hood, corrupting the values.
>> +	 */
>> +	clar__assert_equal(CLAR_CURRENT_FILE, CLAR_CURRENT_FUNC,
>> +			   CLAR_CURRENT_LINE,
>> +			   "expect_result != result", 1,
>> +			   "%"PRIuMAX, expect_result, result);
>> +}
> 
> This was an exciting bug to track down. If you use i_fmt() here, you get
> some neat undefined behavior. It worked for gcc, but failed with clang
> (but only with -O2!).
> 
> Obviously this was me using it wrong, and the "i" in the macro should
> have been a hint. But this invocation is kind of ugly, with the explicit
> mentions of internal CLAR variables. clar__assert_equal() understands
> PRIuMAX as a comparator, but there doesn't appear to be any macro to use
> it nicely.
> 
> Should there be a generic cl_assert_equal() that fills in the first
> few parameters but is otherwise type-agnostic?

Patrick's got a PR open for that at 
https://github.com/clar-test/clar/pull/117 it seems to have got stuck 
because of a lack of review.

>    # start of suite 10: parse_int
>    not ok 59 - parse_int::basic
>        ---
>        reason: |
>          expect_result != result
>          10 != 11
>        at:
>          file: 't/unit-tests/u-parse-int.c'
>          line: 41
>          function: 'test_parse_int__basic'
>        ---
> 
> OK, but "prove t/unit-tests/bin/unit-tests" gives me:
> 
>    t/unit-tests/bin/unit-tests .. Failed 1/59 subtests
>    
>    Test Summary Report
>    -------------------
>    t/unit-tests/bin/unit-tests (Wstat: (none) Tests: 59 Failed: 1)
>      Failed test:  59
>      Parse errors: Badly formed hash line: '---' at /usr/share/perl/5.40/TAP/Parser/YAMLish/Reader.pm line 244.
> 
> Yuck. It actually does have what I need (that test 59 was the failure),
> so the extra parse error is mostly a red herring (though it does prevent
> us finding any further failures). I think in TAP that arbitrary output
> is supposed to be prefixed with a "#".

TAP also allows you to embed YAML and unfortunately that's what clar 
tries to do but that last "    ---" line should be "    ...". With the 
diff below (which I'm afraid thunderbird will probably mangle) prove 
parses the output correctly but still does not print the error message. 
I'll update clar's self tests and open a PR later this week.

---- 8< ----
diff --git a/t/unit-tests/clar/clar/print.h b/t/unit-tests/clar/clar/print.h
index 89b66591d75..6a2321b399d 100644
--- a/t/unit-tests/clar/clar/print.h
+++ b/t/unit-tests/clar/clar/print.h
@@ -164,7 +164,7 @@ static void clar_print_tap_ontest(const char 
*suite_name, const char *test_name,
                          printf("      file: '"); 
print_escaped(error->file); printf("'\n");
                          printf("      line: %" PRIuMAX "\n", 
error->line_number);
                          printf("      function: '%s'\n", error->function);
-                        printf("    ---\n");
+                        printf("    ...\n");
                  }

                  break;
---- >8 ----

> In test-lib.sh, we solve this by
> only allowing "--verbose-log", not regular "-v", under a TAP harness.
> 
> I kind of wonder if we should have t0011-unit-tests.sh that simply runs
> unit-tests and filters the output into stdout and stderr.

I don't have a strong opinion on this but now that we don't run the unit 
tests in parallel because clar links them all into a single executable 
there is less reason to use prove.

Thanks

Phillip