From: Ivan Kanis <expire-by-2010-08-14@kanis•fr>
To: Dmitry Potapov <dpotapov@gmail•com>
Cc: Nguyen Thai Ngoc Duy <pclouds@gmail•com>,
jaredhance@gmail•com, Avery Pennarun <apenwarr@gmail•com>,
jnareb@gmail•com, git <git@vger•kernel.org>
Subject: Re: Excessive mmap [was Git server eats all memory]
Date: Mon, 09 Aug 2010 18:34:29 +0200 [thread overview]
Message-ID: <westyn3n3sa.fsf@kanis.fr> (raw)
In-Reply-To: <AANLkTi=6JcwLuyNWq9oYzE_E+7DSn-sEvR-X3AHvXM6U@mail.gmail.com> (Dmitry Potapov's message of "Mon, 9 Aug 2010 16:35:11 +0400")
Dmitry Potapov <dpotapov@gmail•com> wrote:
> Though Git uses MAP_PRIVATE with mmap, this only marks mapped pages as
> copy-on-write. Because cloning does not change the pack file, all those
> pages should be shared.
I reran the test today. One client is receiving while the other one is
compressing. I have to interrupt both client because the server is
becoming unusable. I am sure you are right about sharing the pages but I
can't test it.
> On 64-bit architecture, you have plenty virtual space, and mapping
> a file to memory should not take much physical memory (only space
> needed for system tables).
What I can tell from the mmap man page is that it should map memory to a
file. I assume it shouldn't take up physical memory. However I am seeing
physical memory being consumed. It might be a feature of the kernel. Is
there a way to turn it off?
> You can reduce core.packedGitWindowSize in the Git configuration to
> see if it helps, but I doubt that it will have any noticeable effect.
It was worth a shot, it didn't help...
Looking some more into it today the bulk of the memory allocation
happens in write_pack_file in the following loop.
for (; i < nr_objects; i++) {
if (!write_one(f, objects + i, &offset))
break;
display_progress(progress_state, written);
}
This eventually calls write_object, here I am wondering if the
unuse_pack function is doing its job. As far as I can tell it writes a
null in memory, that I think is not enough to reclaim memory.
I also looked at the use_pack function where the mmap is
happening. Would it be worth refactoring this function so that it uses
an index withing a file instead of mmap?
Unless I hear of a better idea I'll be trying that tomorrow...
Take care,
--
Ivan Kanis
I can stand brute force, but brute reason is quite unbearable. There
is something unfair about its use. It is hitting below the intellect.
-- Oscar Wilde
next prev parent reply other threads:[~2010-08-09 16:35 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-08-04 14:57 Git server eats all memory Ivan Kanis
2010-08-04 15:55 ` Matthieu Moy
2010-08-04 17:50 ` Ivan Kanis
2010-08-04 20:12 ` Avery Pennarun
2010-08-05 6:33 ` Ivan Kanis
2010-08-05 22:45 ` Jared Hance
2010-08-06 1:37 ` Nguyen Thai Ngoc Duy
2010-08-06 1:51 ` Nguyen Thai Ngoc Duy
2010-08-06 11:34 ` Jakub Narebski
2010-08-06 17:23 ` Ivan Kanis
2010-08-07 6:42 ` Dmitry Potapov
2010-08-09 10:12 ` Excessive mmap [was Git server eats all memory] Ivan Kanis
2010-08-09 12:35 ` Dmitry Potapov
2010-08-09 16:34 ` Ivan Kanis [this message]
2010-08-09 16:50 ` Avery Pennarun
2010-08-09 17:45 ` Tomas Carnecky
2010-08-09 18:17 ` Avery Pennarun
2010-08-09 21:28 ` Dmitry Potapov
2010-08-11 15:47 ` Ivan Kanis
2010-08-11 16:35 ` Avery Pennarun
[not found] ` <wes4oetv31i.fsf@kanis.fr>
2010-08-17 17:07 ` Dmitry Potapov
2018-06-20 14:53 ` Duy Nguyen
[not found] ` <AANLkTi=yeTh2tKn9t_=iZbdB5VLrfCPZ2_fBpYdf9wta@mail.gmail.com>
[not found] ` <wesbp9cnnag.fsf@kanis.fr>
2010-08-09 9:57 ` Git server eats all memory Nguyen Thai Ngoc Duy
2010-08-09 17:38 ` Ivan Kanis
2010-08-10 0:46 ` Robin H. Johnson
2010-08-10 2:31 ` Sverre Rabbelier
2010-08-11 10:30 ` Sam Vilain
2010-08-11 15:54 ` Ivan Kanis
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=westyn3n3sa.fsf@kanis.fr \
--to=expire-by-2010-08-14@kanis$(echo .)fr \
--cc=apenwarr@gmail$(echo .)com \
--cc=dpotapov@gmail$(echo .)com \
--cc=git@vger$(echo .)kernel.org \
--cc=jaredhance@gmail$(echo .)com \
--cc=jnareb@gmail$(echo .)com \
--cc=pclouds@gmail$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox