Subject: Re: Fast access git-rev-list output: some OS knowledge required

From: Shawn Pearce <[email-masked]>
Date: Wed, 6 Dec 2006 15:18:16 -0500

Marco Costalba <[email-masked]> wrote:
> On 12/6/06, Shawn Pearce <[email-masked]> wrote:
> >Shawn Pearce <[email-masked]> wrote:
> >
> >Perhaps there is some fast IPC API supported by Qt that you could
> >use to run the revision listing outside of the main UI process,
> >to eliminate the bottlenecks you are seeing and remove the problems
> >noted above?  One that doesn't involve reading from a pipe I mean...
> >
> 
> Qt it's very fast in reading from files, also git-rev-list is fast in
> write to a file...the problem is I would not want the file to be saved
> on disk, but stay cached in the OS memory for the few seconds needed
> to be written and read back, and then deleted. It's a kind of shared
> memory at the end. But I don't know how to realize it.

On a modern Linux (probably your largest target audience) a small
file which has a very short lifespan (few seconds) is unlikey to
hit the platter.  Most filesystems will put the data into buffer
cache and delay writing to disk because temporary files are so
common on UNIX.

Though our resident Linux experts may chime in with more details...
 
> Also let git-rev-list to write directly in qgit process address space
> would be nice, indeed very nice.

And ugly.  :-)

SysV IPC (shared memory, semaphores) are messy and difficult to
get right.  mmap against a random file in the filesystem tends
to work better on those systems which support it well, provided
that the file isn't on a network mount.  But again you still need
semaphores or something like them to control access to the data in
the mmap'd region.

I was thinking that maybe if Qt had a bounded buffer available for
use between a process and its child, that you could use that to run
your own "qgit-rev-list" child and get the data back more quickly,
without the need for a temporary file.  But it doesn't look like
they have one.  Oh well.

Your current temporary file approach is probably the best you can
get, and has the simplest possible implementation.  Doing better
would require linking against libgit.a, and getting the core Git
hackers to make at least the revision machinery more useful in a
library setting.

--