Subject: [PATCH 0/4] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

From: Taylor Blau <[email-masked]>
Date: Sun, 1 Sep 2024 12:03:15 -0400

This series adds a build-time knob to allow selecting an alternative
SHA-1 implementation for non-cryptographic hashing within Git, starting
with the `hashwrite()` family of functions.

This series is the result of starting to roll out verbatim multi-pack
reuse within GitHub's infrastructure. I noticed that on larger
repositories, it is harder thus far to measure a CPU speed-up on clones
where multi-pack reuse is enabled.

After some profiling, I noticed that we spend a significant amount of
time in hashwrite(), which is not all that surprising. But much of that
time is wasted in GitHub's infrastructure, since we are using the same
collision-detecting SHA-1 implementation to produce a trailing checksum
for the pack which does not need to be cryptographically secure.

This series teaches a new set of build-time knobs: OPENSSL_SHA1_FAST,
BLK_SHA1_FAST, and APPLE_COMMON_CRYPTO_SHA1_FAST, which can be used to
select an alterantive SHA-1 implementation for non-cryptographic uses
within Git.

The series is laid out as follows:

  - The first two patches are preparatory, allowing us to include
    multiple SHA-1 wrapper headers and adding scaffolding functions for
    the _fast() variants, respectively.

  - The third patch introduces the build-time knobs for selecting which
    SHA-1 implementation is used for non-cryptographic purposes.

  - The fourth and final patch updates the hashwrite() implementation to
    use the _fast() variants.

>From the final commit, a clone of the kernel which took 19.219 seconds
on my machine with no build-time modifications now only takes 11.597
seconds when compiling with OPENSSL_SHA1_FAST=1, for a ~39.7% speed-up.
On a freshly-repacked copy of git.git, the time goes from 758ms to 333ms
for a ~56% speed-up.

Thanks in advance for your review!

Taylor Blau (4):
  sha1: do not redefine `platform_SHA_CTX` and friends
  hash.h: scaffolding for _fast hashing variants
  Makefile: allow specifying a SHA-1 for non-cryptographic uses
  csum-file.c: use fast SHA-1 implementation when available

 Makefile          | 25 ++++++++++++++++++
 block-sha1/sha1.h |  2 ++
 csum-file.c       | 18 ++++++-------
 hash.h            | 67 +++++++++++++++++++++++++++++++++++++++++++++++
 object-file.c     | 42 +++++++++++++++++++++++++++++
 sha1/openssl.h    |  2 ++
 sha1dc_git.h      |  3 +++
 7 files changed, 150 insertions(+), 9 deletions(-)

base-commit: 159f2d50e75c17382c9f4eb7cbda671a6fa612d1
-- 
2.46.0.425.ge8f5cbd280c