Subject: Re: Git is not scalable with too many refs/*

From: Martin Fick <[email-masked]>
Date: Mon, 26 Sep 2011 15:52:04 -0600

On Monday, September 26, 2011 03:39:33 pm Martin Fick wrote:
> On Monday, September 26, 2011 02:28:53 pm Julian Phillips
> wrote:
> > >> Random thought.  What happens to the with
> > >> compression case if you leave the commit in, but
> > >> add a sleep(15) to the end of sort_refs_list?
> > > 
> > > Why, what are you thinking?  Hmm, I am trying this on
> > > the non gced repo and it doesn't seem to be
> > > completing (no cpu usage)!  It appears that perhaps
> > > it is being called many times (the sleeping would
> > > explain no cpu usage)?!?  This could be a real
> > > problem, this should only get called once right?
> > 
> > I was just wondering if the time taken to get the refs
> > was changing the interaction with something else.  Not
> > very likely, but ...
> > 
> > I added a print statement, and it was called four times
> > when I had unpacked refs, and once with packed.  So,
> > maybe you are hitting some nasty case with unpacked
> > refs.  If you use a print statement instead of a sleep,
> > how many times does sort_refs_lists get called in your
> > unpacked case?  It may well also be worth calculating
> > the time taken to do the sort.
> 
> In my case it was called 18785 times!  Any other tests I
> should run?

Gerrit stores the changes in directories under refs/changes 
named after the last 2 digits of the change.  Then under 
each change it stores each patchset.  So it looks like this:   
refs/changes/dd/change_num/ps_num

I noticed that:

 ls refs/changes/* | wc -l 
 -> 18876

somewhat close, but not super close to 18785,  I am not sure 
if that is a clue.  It's almost like each change is causing 
a re-sort,

-Martin

-- 
Employee of Qualcomm Innovation Center, Inc. which is a 
member of Code Aurora Forum