From: Luke Diamand <luke@diamand•org>
To: Pete Wyckoff <pw@padd•com>
Cc: Corey Thompson <cmtptr@gmail•com>, git@vger•kernel.org
Subject: Re: git-p4 out of memory for very large repository
Date: Mon, 02 Sep 2013 20:42:36 +0100 [thread overview]
Message-ID: <5224EA2C.7090001@diamand.org> (raw)
In-Reply-To: <20130829224609.GB25879@padd.com>
I guess you could try changing the OOM score for git-fast-import.
change /proc/<pid>/oomadj.
I think a value of -31 would make it very unlikely to be killed.
On 29/08/13 23:46, Pete Wyckoff wrote:
> cmtptr@gmail•com wrote on Wed, 28 Aug 2013 11:41 -0400:
>> On Mon, Aug 26, 2013 at 09:47:56AM -0400, Corey Thompson wrote:
>>> You are correct that git-fast-import is killed by the OOM killer, but I
>>> was unclear about which process was malloc()ing so much memory that the
>>> OOM killer got invoked (as other completely unrelated processes usually
>>> also get killed when this happens).
>>>
>>> Unless there's one gigantic file in one change that gets removed by
>>> another change, I don't think that's the problem; as I mentioned in
>>> another email, the machine has 32GB physical memory and the largest
>>> single file in the current head is only 118MB. Even if there is a very
>>> large transient file somewhere in the history, I seriously doubt it's
>>> tens of gigabytes in size.
>>>
>>> I have tried watching it with top before, but it takes several hours
>>> before it dies. I haven't been able to see any explosion of memory
>>> usage, even within the final hour, but I've never caught it just before
>>> it dies, either. I suspect that whatever the issue is here, it happens
>>> very quickly.
>>>
>>> If I'm unable to get through this today using the incremental p4 sync
>>> method you described, I'll try running a full-blown clone overnight with
>>> top in batch mode writing to a log file to see whether it catches
>>> anything.
>>>
>>> Thanks again,
>>> Corey
>>
>> Unforunately I have not made much progress. The incremental sync method
>> fails with the output pasted below. The change I specified is only one
>> change number above where that repo was cloned...
>
> I usually just do "git p4 sync @505859". The error message below
> crops up when things get confused. Usually after a previous
> error. I tend to destroy the repo and try again. Sorry I don't
> can't explain better what's happening here. It's not a memory
> issue; it reports only 24 MB used.
>
>> So I tried a 'git p4 rebase' overnight with top running, and as I feared
>> I did not see anything out of the ordinary. git, git-fast-import, and
>> git-p4 all hovered under 1.5% MEM the entire time, right up until
>> death. The last entry in my log shows git-fast-import at 0.8%, with git
>> and git-p4 at 0.0% and 0.1%, respectively. I could try again with a
>> more granular period, but I feel like this method is ultimately a goose
>> chase.
>
> Bizarre. There is no good explanation why memory usage would go
> up to 32 GB (?) within one top interval (3 sec ?). My theory
> about one gigantic object is debunked: you have only the 118 MB
> one. Perhaps there's some container or process memory limit, as
> Luke guessed, but it's not obvious here.
>
> The other big hammer is "strace". If you're still interested in
> playing with this, you could do:
>
> strace -vf -tt -s 200 -o /tmp/strace.out git p4 clone ....
>
> and hours later, see if something suggests itself toward the
> end of that output file.
>
> -- Pete
next prev parent reply other threads:[~2013-09-02 19:42 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-08-23 1:12 git-p4 out of memory for very large repository Corey Thompson
2013-08-23 7:16 ` Luke Diamand
2013-08-23 11:48 ` Corey Thompson
2013-08-23 11:59 ` Corey Thompson
2013-08-23 19:42 ` Luke Diamand
2013-08-24 0:56 ` Corey Thompson
2013-08-25 15:50 ` Pete Wyckoff
2013-08-26 13:47 ` Corey Thompson
2013-08-28 15:41 ` Corey Thompson
2013-08-29 22:46 ` Pete Wyckoff
2013-09-02 19:42 ` Luke Diamand [this message]
2013-09-06 19:03 ` Corey Thompson
2013-09-07 8:19 ` Pete Wyckoff
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5224EA2C.7090001@diamand.org \
--to=luke@diamand$(echo .)org \
--cc=cmtptr@gmail$(echo .)com \
--cc=git@vger$(echo .)kernel.org \
--cc=pw@padd$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox