Subject: Re: [PATCH v3 25/25] p5326: perf tests for MIDX bitmaps

From: Jeff King <[email-masked]>
Date: Thu, 12 Aug 2021 17:18:52 -0400

On Tue, Jul 27, 2021 at 05:20:28PM -0400, Taylor Blau wrote:

> These new performance tests demonstrate effectively the same behavior as
> p5310, but use a multi-pack bitmap instead of a single-pack one.
> 
> Notably, p5326 does not create a MIDX bitmap with multiple packs. This
> is so we can measure a direct comparison between it and p5310. Any
> difference between the two is measuring just the overhead of using MIDX
> bitmaps.
> 
> Here are the results of p5310 and p5326 together, measured at the same
> time and on the same machine (using a Xenon W-2255 CPU):

Neat. I think having separate perf regression tests for regular and mix
bitmaps will be useful, but being able to compare the pack and mix
versions is a cherry on top.

There was one funny number:

>     5310.2: repack to disk                                96.78(93.39+11.33)
>     5326.2: setup multi-pack index                        78.99(75.29+11.58)

In p5310, that step is repacking and writing bitmaps. With the midx,
it's repacking, then writing a midx with bitmaps. I'd expect the latter
to be strictly slower than the former, but here it's faster.

Running the code locally, I got similar results (with p5310 just a tiny
bit faster). So it may have just been noise or some other timing issue.

  As an aside, I think that test is a little bit bogus due to
  GIT_PERF_REPEAT_COUNT; the first trial will generate bitmaps from
  scratch, and then subsequent runs will reuse partial results. It
  probably should "rm -f .git/objects/*.bitmap" within the test. We can
  deal with that separately, though.

-Peff