Subject: Re: [PATCH 1/9] docs: update pack index v3 format

From: "brian m. carlson" <[email-masked]>
Date: Sat, 20 Sep 2025 15:23:07 +0000

On 2025-09-19 at 22:08:03, Junio C Hamano wrote:
> "brian m. carlson" <[email-masked]> writes:
> 
> > Our current pack index v3 format uses 4-byte integers to find the
> > trailer of the file.  This effectively means that the file cannot be
> > much larger than 2^32.  While this might at first seem to be okay, we
> > expect that each object will have at least 64 bytes worth of data, which
> > means that no more than about 67 million objects can be stored.
> >
> > Again, this might seem fine, but unfortunately, we know of many users
> > who attempt to create repos with extremely large numbers of commits to
> > get a "high score," and we've already seen repositories with at least 55
> > million commits.  In the interests of gracefully handling repositories
> > even for these well-intentioned but ultimately misguided users, let's
> > change these lengths to 8 bytes.
> 
> Very sensible.
> 
> I do also agree that 32-byte is the natural size for the trailing
> hash, but I found that the two paragraphs below was far more than
> necessary.  As they argue, we use a truncated hash anywhere in our
> file formats, so I would have understood if the explanation were
> 
>     "20" in "A copy of the 20-byte SHA-256 checksum" is an obvious
>     typo, as SHA-256 is longer than that.  Fix it to "32".
> 
> instead of these two paragraphs.
> 
> Or did we mean to use a truncated hash back when this transition
> design was proposed originally?

I think we intended to use a 20-byte value originally because we felt we
didn't need the full 32 bytes for an index or pack checksum.  However,
as I mentioned, we use the 32-byte checksum for SHA-256 already, so all
it does is add complexity to try to mandate a 20-byte value.
-- 
brian m. carlson (they/them)
Toronto, Ontario, CA