From: Ivan Kanis <expire-by-2010-08-14@kanis•fr>
To: Dmitry Potapov <dpotapov@gmail•com>
Cc: Ivan Kanis <expire-by-2010-08-11@kanis•fr>,
Nguyen Thai Ngoc Duy <pclouds@gmail•com>,
jaredhance@gmail•com, Avery Pennarun <apenwarr@gmail•com>,
jnareb@gmail•com, git <git@vger•kernel.org>
Subject: Excessive mmap [was Git server eats all memory]
Date: Mon, 09 Aug 2010 12:12:34 +0200 [thread overview]
Message-ID: <wesy6cgm6wd.fsf_-_@kanis.fr> (raw)
In-Reply-To: <AANLkTinyX9cABkEDy3HBZoDVNWos2djNBSaw2Hg_yzAO@mail.gmail.com> (Dmitry Potapov's message of "Sat, 7 Aug 2010 10:42:08 +0400")
Dmitry Potapov <dpotapov@gmail•com> wrote:
> On Fri, Aug 06, 2010 at 07:23:17PM +0200, Ivan Kanis wrote:
>>
>> I expected the malloc to take 4G but was surprised it didn't. It seems
>> to be mmap taking all the memory. I am not familiar with that function,
>> it looks like it's mapping memory to a file... Is it reasonable to mmap
>> so much memory?
>
> AFAIK, Git does not need to mmap the whole pack to memory, but it
> is more efficient to mmap the whole pack wherever possible, because
> it has a completely random access, so if you store only one sliding
> window, you will have to re-read it many times. Besides, mmap size
> does not mean that so much physical memory is used. Pages should
> be loaded when they are necessary, and if you have more than one
> client cloning the same repo, this memory should be shared by them.
I have clone identical repositories and the system starts to swap. I
think it shows that cloning two repository doesn't share mmap.
I saw this constant defined in git-compat-util.h
/* This value must be multiple of (pagesize * 2) */
#define DEFAULT_PACKED_GIT_WINDOW_SIZE \
(sizeof(void*) >= 8 \
? 1 * 1024 * 1024 * 1024 \
: 32 * 1024 * 1024)
If I read this correctly git is allocating 1G of mmap on 64 bit
architecture. Isn't that a bit much? I am running on a 64 bit server so
I have bumped DEFAULT_PACKED_GIT_WINDOW_SIZE down to 64M but, alas, the
pack command still takes over 4G...
I'll keep investigating today,
Take care,
--
Ivan Kanis
Nothing in life is to be feared. It is only to be understood.
-- Marie Curie
next prev parent reply other threads:[~2010-08-09 10:13 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-08-04 14:57 Git server eats all memory Ivan Kanis
2010-08-04 15:55 ` Matthieu Moy
2010-08-04 17:50 ` Ivan Kanis
2010-08-04 20:12 ` Avery Pennarun
2010-08-05 6:33 ` Ivan Kanis
2010-08-05 22:45 ` Jared Hance
2010-08-06 1:37 ` Nguyen Thai Ngoc Duy
2010-08-06 1:51 ` Nguyen Thai Ngoc Duy
2010-08-06 11:34 ` Jakub Narebski
2010-08-06 17:23 ` Ivan Kanis
2010-08-07 6:42 ` Dmitry Potapov
2010-08-09 10:12 ` Ivan Kanis [this message]
2010-08-09 12:35 ` Excessive mmap [was Git server eats all memory] Dmitry Potapov
2010-08-09 16:34 ` Ivan Kanis
2010-08-09 16:50 ` Avery Pennarun
2010-08-09 17:45 ` Tomas Carnecky
2010-08-09 18:17 ` Avery Pennarun
2010-08-09 21:28 ` Dmitry Potapov
2010-08-11 15:47 ` Ivan Kanis
2010-08-11 16:35 ` Avery Pennarun
[not found] ` <wes4oetv31i.fsf@kanis.fr>
2010-08-17 17:07 ` Dmitry Potapov
2018-06-20 14:53 ` Duy Nguyen
[not found] ` <AANLkTi=yeTh2tKn9t_=iZbdB5VLrfCPZ2_fBpYdf9wta@mail.gmail.com>
[not found] ` <wesbp9cnnag.fsf@kanis.fr>
2010-08-09 9:57 ` Git server eats all memory Nguyen Thai Ngoc Duy
2010-08-09 17:38 ` Ivan Kanis
2010-08-10 0:46 ` Robin H. Johnson
2010-08-10 2:31 ` Sverre Rabbelier
2010-08-11 10:30 ` Sam Vilain
2010-08-11 15:54 ` Ivan Kanis
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=wesy6cgm6wd.fsf_-_@kanis.fr \
--to=expire-by-2010-08-14@kanis$(echo .)fr \
--cc=apenwarr@gmail$(echo .)com \
--cc=dpotapov@gmail$(echo .)com \
--cc=expire-by-2010-08-11@kanis$(echo .)fr \
--cc=git@vger$(echo .)kernel.org \
--cc=jaredhance@gmail$(echo .)com \
--cc=jnareb@gmail$(echo .)com \
--cc=pclouds@gmail$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox