Subject: Re: [ANNOUNCE] Example Cogito Addon - cogito-bundle

From: Shawn Pearce <[email-masked]>
Date: Wed, 18 Oct 2006 17:32:26 -0400

Linus Torvalds <[email-masked]> wrote:
> On Wed, 18 Oct 2006, Shawn Pearce wrote:
> >
> > I guess its my turn then to work in the mmap window code, huh?  :-)
> 
> There are bigger reasons to _never_ allow packs to contain deltas to 
> outside of themselves:
> 
>  - there's no point. 
>  - it's a bad design. 

That and all of the other reasons you cited in your message are
why I haven't finished trying to use some sort of dictionary based
compression for packing objects.

On the other hand we've already seen how packs >1.5 GiB in size
(certainly well within the 4 GiB limitation in the current index
file format) cannot be repacked by git-repack-objects on a 32
bit address space as the entire pack file is mmap'd on one shot.
After the kernel space of ~1 GiB and the pack file at ~1.5 GiB
there's very little address space left for the application code.

My comment that you quoted was about mmap'ing the pack files in
large chunks (around 64-128 MiB at a time, but configurable from
.git/config) rather than as an entire massive mapping.  It had
absolutely nothing to do about changing the pack file format, the
index format, or any other on disk format.  Although it would add
a new pair of configuration options to .git/config.  Is that change
too radical?  :-)

With such a change the Git and Linux kernel repositories would both
still mmap in one chunk but much larger projects like Mozilla or
very large pack files coming out of git-fastimport would actually
be usable on 32 bit architectures without running into address space
limitations so quickly.  Git would also be slightly more usable for
some people who have a lot of very uncompressable data stored in Git.

Unless of course you are actively working on a fix for the Linux
kernel so that we can actually have all 4 GiB of virtual address
space available for the userspace git-repack-objects process.
Or have some sort of secret plan to upgrade everyone who uses Git
to 64 bit processors which support 64 bit address spaces...

-- 
Shawn.