From: Junio C Hamano <gitster@pobox•com>
To: Stefan Beller <sbeller@google•com>
Cc: Michael Haggerty <mhagger@alum•mit.edu>,
"git\@vger.kernel.org" <git@vger•kernel.org>
Subject: Re: [PATCH 0/3] Another approach to large transactions
Date: Mon, 20 Apr 2015 15:51:02 -0700 [thread overview]
Message-ID: <xmqqzj62ifc9.fsf@gitster.dls.corp.google.com> (raw)
In-Reply-To: <CAGZ79kZWm=Mi6o4jMNthiDRcR9irs_5MyRuEmHdDSrn-JFpQ=g@mail.gmail.com> (Stefan Beller's message of "Mon, 20 Apr 2015 15:26:56 -0700")
Stefan Beller <sbeller@google•com> writes:
> The problem comes from guessing the number of fds we're allowed to use.
> At first I thought it was a fundamental issue with the code being broken, but
> it turns out we just need a larger offset as we apparently have 9 files open
> already, before the transaction even starts.
> I did not expect the number to be that high, which is why I came up with the
> arbitrary number of 8 (3 for stdin/out/err, maybe packed refs and reflog so I
> guessed, 8 would do fine).
>
> I am not sure if the 9 is a constant or if it scales to some unknown
> property yet.
> So to make the series work, all we need is:
>
> - int remaining_fds = get_max_fd_limit() - 8;
> + int remaining_fds = get_max_fd_limit() - 9;
>
> I am going to try to understand where the 9 comes from and resend the patches.
I have a suspicion that the above is an indication that the approach
is fundamentally not sound. 9 may be OK in your test repository,
but that may fail in a repository with different resource usage
patterns.
On the core management side, xmalloc() and friends retry upon
failure, after attempting to free the resource. I wonder if your
codepath can do something similar to that, perhaps?
On the other hand, it may be that this "let's keep it open as long
as possible, as creat-close-open-write-close is more expensive" may
not be worth the complexity. I wonder if it might not be a bad idea
to start with a simpler rule, e.g. "use creat-write-close for ref
updates outside transactions, and creat-close-open-write-close for
inside transactions, as that is likely to be multi-ref updates" or
something stupid and simple like that?
Michael?
next prev parent reply other threads:[~2015-04-20 22:51 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-04-16 23:17 [PATCH 0/3] Another approach to large transactions Stefan Beller
2015-04-16 23:17 ` [PATCH 1/3] refs.c: remove lock_fd from struct ref_lock Stefan Beller
2015-04-16 23:17 ` [PATCH 2/3] Move get_max_fd_limit(void) to git_compat_util.h Stefan Beller
2015-04-16 23:17 ` [PATCH 3/3] refs.c: enable large transactions Stefan Beller
2015-04-17 17:09 ` [PATCH 0/3] Another approach to " Junio C Hamano
2015-04-17 22:12 ` Junio C Hamano
2015-04-17 22:17 ` Stefan Beller
2015-04-17 23:31 ` Stefan Beller
2015-04-20 22:26 ` Stefan Beller
2015-04-20 22:51 ` Junio C Hamano [this message]
2015-04-20 23:07 ` Stefan Beller
2015-04-21 0:31 ` Stefan Beller
2015-04-21 0:35 ` [PATCH] refs.c: enable " Stefan Beller
2015-04-21 17:16 ` Junio C Hamano
2015-04-21 17:24 ` Stefan Beller
2015-04-21 18:00 ` Junio C Hamano
2015-04-21 19:06 ` [PATCHv2] " Stefan Beller
2015-04-21 19:56 ` Stefan Beller
2015-04-22 14:11 ` Michael Haggerty
2015-04-22 19:09 ` Stefan Beller
2015-04-22 20:12 ` Michael Haggerty
2015-04-21 17:22 ` [PATCH] " Junio C Hamano
2015-04-21 23:21 ` [PATCH 0/3] Another approach to " Jeff King
2015-04-22 19:14 ` Stefan Beller
2015-04-22 20:11 ` Jeff King
2015-04-21 17:19 ` Junio C Hamano
2015-04-21 17:31 ` Stefan Beller
2015-04-21 12:37 ` Michael Haggerty
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=xmqqzj62ifc9.fsf@gitster.dls.corp.google.com \
--to=gitster@pobox$(echo .)com \
--cc=git@vger$(echo .)kernel.org \
--cc=mhagger@alum$(echo .)mit.edu \
--cc=sbeller@google$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox