Subject: Re: Git is not scalable with too many refs/*

From: Martin Fick <[email-masked]>
Date: Mon, 26 Sep 2011 18:12:31 -0600

On Monday, September 26, 2011 05:26:55 pm Julian Phillips 
wrote:
> On Mon, 26 Sep 2011 15:52:04 -0600, Martin Fick wrote:
> > On Monday, September 26, 2011 03:39:33 pm Martin Fick 
wrote:
> >> On Monday, September 26, 2011 02:28:53 pm Julian
> >> Phillips
> >> In my case it was called 18785 times!  Any other tests
> >> I should run?
> > 
> > Gerrit stores the changes in directories under
> > refs/changes named after the last 2 digits of the
> > change.  Then under each change it stores each
> > patchset.  So it looks like this:
> > refs/changes/dd/change_num/ps_num
> > 
> > I noticed that:
> >  ls refs/changes/* | wc -l
> >  -> 18876
> > 
> > somewhat close, but not super close to 18785,  I am not
> > sure if that is a clue.  It's almost like each change
> > is causing a re-sort,
> 
> basically, it is ...
> 
> Back when I made that change, I failed to notice that
> get_ref_dir was recursive for subdirectories ... sorry
> ...
> 
> Hopefully this should speed things up.  My test repo went
> from ~17m user time, to ~2.5s.
> Packing still make things much faster of course.

Excellent!  This works (almost, in my refs.c it is called 
sort_ref_list, not sort_refs_list).  So, on the non garbage 
collected repo, git branch now takes ~.5s, and in the 
garbage collected one it takes only ~.05s!

Thanks way much!!!

-Martin

-- 
Employee of Qualcomm Innovation Center, Inc. which is a 
member of Code Aurora Forum