Subject: Re: [PATCH 4/5] meson: wire up benchmarks

From: Toon Claes <[email-masked]>
Date: Thu, 10 Apr 2025 13:44:55 +0200

Patrick Steinhardt <[email-masked]> writes:

> Wire up benchmarks in Meson. The setup is mostly the same as how we wire
> up our tests. The only difference is that benchmarks get wired up via
> the `benchmark()` option instead of via `test()`, which gives them a bit
> of special treatment:
>
>   - Benchmarks never run in parallel.
>
>   - Benchmarks aren't run by default when tests are executed.
>
>   - Meson does not inject the `MALLOC_PERTURB` environment variable.
>
> Using benchmarks is quite simple:
>
>     ```
>     $ meson setup build
>     # Run all benchmarks.
>     $ meson test -C build --benchmark
>     # Run a specific benchmark.
>     $ meson test -C build --benchmark p0000-*
>     ```

I really appreciate the efforts, and I'd love to run the perf tests with
Meson, but at the moment I don't know how these changes are useful
because this doesn't print anything relevant:

    $ meson test -C build --benchmark 'p0005-*'
    ninja: Entering directory `/home/toon/devel/git/build'
    [1/28] Generating GIT-VERSION-FILE with a custom command (wrapped by meson to set env)
    1/1 p0005-status        OK              7.39s

    Ok:                 1
    Expected Fail:      0
    Fail:               0
    Unexpected Pass:    0
    Skipped:            0
    Timeout:            0

    Full log written to /home/toon/devel/git/build/meson-logs/testlog.txt

> Other than that the usual command line arguments accepted when running
> tests are also accepted when running benchmarks.
>
> Note that the benchmarking target is somewhat limited because it will
> only run benchmarks for the current build. Other usecases, like running
> benchmarks against multiple different versions of Git, are not currently
> supported. Users should continue to use "t/perf/run" for those usecases.
> The script should get extended at one point in time to support Meson,
> but this is outside of the scope of this series.

Yeah, this is unfortunate, but totally understandable. I've been digging
in `t/perf/run` and `f/perf/aggregate.perl` and it doesn't look easy to
adapt to Meson.

But I was wondering, instead of trying to fully integrate Meson into
those scripts, could we modify the scripts so they work with binaries
built by Meson? I mean, if we could run
`cd t/perf && ./run ../../build1 ../../build2 p0005*` and it would
simply run the benchmarks in those directories (without trying to check
out code and build the sources). I think this would help a lot already.

-- 
Toon