From: "Shawn O. Pearce" <spearce@spearce•org>
To: Junio C Hamano <gitster@pobox•com>
Cc: Marco Costalba <mcostalba@gmail•com>, git@vger•kernel.org
Subject: Re: [PATCH 03/11] Use new compress helpers in fast-import
Date: Sun, 3 Feb 2008 20:48:50 -0500 [thread overview]
Message-ID: <20080204014849.GG24004@spearce.org> (raw)
In-Reply-To: <7v3as9slc6.fsf@gitster.siamese.dyndns.org>
Junio C Hamano <gitster@pobox•com> wrote:
> Marco Costalba <mcostalba@gmail•com> writes:
>
> > Here is slightly more difficult, in particular
> > a xrealloc() has been substituted with a
> > free() + xmalloc() to keep the code simple.
> >
> > Signed-off-by: Marco Costalba <mcostalba@gmail•com>
> > ---
> > fast-import.c | 45 +++++++++++++++------------------------------
> > 1 files changed, 15 insertions(+), 30 deletions(-)
>
> I'll let Shawn comment on this. The realloc() does not seem to
> be using the contents in the buffer from the previous round, so
> I suspect that a free() followed by an independent alloc() would
> be an improvement when the later call uses much larger buffer
> than the previous one, but would be a waste if the later one
> needs smaller buffer.
Junio is correct, that xrealloc isn't using the contents of the
buffer from the last round, which makes any memcpy it might do
internally due to movement to a larger buffer an utter waste.
In this new version we are probably always free'ing a buffer of a
much smaller size than we are then later allocating (or in the old
version xrealloc'ing to) because we are switching from a delta to
full content. Its most likely the delta is way smaller, so I'd
guess the malloc implementation is mostly going to another buffer.
In short, Marco's change will most likely do better.
But this is all academic wanking. We're talking about this xrealloc
(or free/xmalloc pair) happening only when we switch packfiles,
which in fast-import is usually every 4 GiB of output. That's a
*lot* of data to write. Who cares how many extra microseconds we
spend to perform this buffer change; we probably hit it only once
every 15-30 minutes, depending on how fast your system is able to
transfer 4 GiB of data out of the source and into a packfile.
--
Shawn.
next prev parent reply other threads:[~2008-02-04 1:49 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-02-02 11:35 [PATCH 01/11] Introduce stream compress helpers Marco Costalba
2008-02-02 11:35 ` [PATCH 02/11] Use new compress helpers in git files Marco Costalba
2008-02-02 11:35 ` [PATCH 03/11] Use new compress helpers in fast-import Marco Costalba
2008-02-02 11:35 ` [PATCH 04/11] Use new compress helpers in http-push.c Marco Costalba
2008-02-02 11:35 ` [PATCH 05/11] Use new compress helpers in sha1_file.c Marco Costalba
2008-02-02 11:35 ` [PATCH 06/11] Better error handling in compress_all() Marco Costalba
2008-02-02 11:35 ` [PATCH 07/11] Introduce stream decompress helpers Marco Costalba
2008-02-02 11:35 ` [PATCH 08/11] Use new decompress_all() helper in git Marco Costalba
2008-02-02 11:35 ` [PATCH 09/11] Convert http-push.c and http-walker.c Marco Costalba
2008-02-02 11:35 ` [PATCH 10/11] Convert builtin-pack/unpack Marco Costalba
2008-02-02 11:35 ` [PATCH 11/11] Convert sha1_file.c to use decompress helpers Marco Costalba
2008-02-04 2:08 ` [PATCH 10/11] Convert builtin-pack/unpack Junio C Hamano
2008-02-04 2:07 ` [PATCH 08/11] Use new decompress_all() helper in git Junio C Hamano
2008-02-04 2:07 ` [PATCH 07/11] Introduce stream decompress helpers Junio C Hamano
2008-02-03 22:54 ` [PATCH 06/11] Better error handling in compress_all() Junio C Hamano
2008-02-03 22:53 ` [PATCH 04/11] Use new compress helpers in http-push.c Junio C Hamano
2008-02-03 22:53 ` [PATCH 03/11] Use new compress helpers in fast-import Junio C Hamano
2008-02-04 1:48 ` Shawn O. Pearce [this message]
2008-02-04 1:41 ` Shawn O. Pearce
2008-02-03 22:54 ` [PATCH 02/11] Use new compress helpers in git files Junio C Hamano
2008-02-03 22:53 ` [PATCH 01/11] Introduce stream compress helpers Junio C Hamano
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20080204014849.GG24004@spearce.org \
--to=spearce@spearce$(echo .)org \
--cc=git@vger$(echo .)kernel.org \
--cc=gitster@pobox$(echo .)com \
--cc=mcostalba@gmail$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox