public inbox for git@vger.kernel.org 
 help / color / mirror / Atom feed

From: Eric Wong <e@80x24•org>
To: git@vger•kernel.org
Subject: [RFC 0/3] dumb HTTP transport speedups
Date: Mon, 11 Jul 2016 20:51:28 +0000	[thread overview]
Message-ID: <20160711205131.1291-1-e@80x24.org> (raw)

TL;DR: dumb HTTP clone from a certain badly-packed repo goes from
~2 hours to ~30 min memory usage drops from 2G to 360M


I hadn't packed the public repo at https://public-inbox.org/git
for a few weeks.  As an admin of a small server limited memory
and CPU resources but fairly good bandwidth, I prefer clients
use dumb HTTP for initial clones.

Unfortunately, I noticed my dinky netbook runs out-of-memory
when using GIT_SMART_HTTP=0 to clone this giant repo; and a
machine with more memory still takes over two hours depending
on network conditions (and uses around 2GB RSS!).

Anyways, https://public-inbox.org/git is better packed, now;
but I've kept https://80x24.org/git-i-forgot-to-pack available
with over 7K loose objects to illustrate the problem:

	(this is dumb HTTP-only)
	git clone --mirror https://80x24.org/git-i-forgot-to-pack

The primary problem is fixed by PATCH 3/3 in this series, and I
can now clone the above in around 30 minutes and "only" seems to
use around 360M memory.

I'll leave git-i-forgot-to-pack up for a few months/year
so others can test and hammer away at it.

The following changes since commit 5c589a73de4394ad125a4effac227b3aec856fa1:

  Third batch of topics for 2.10 (2016-07-06 13:42:58 -0700)

are available in the git repository at:

  git://bogomips.org/git-svn.git dumb-speedups

for you to fetch changes up to b9d5aca4b8e6c9f7fb5ee4e0ce33bb42c4ea2992:

  http-walker: reduce O(n) ops with doubly-linked list (2016-07-11 20:25:51 +0000)

----------------------------------------------------------------
Eric Wong (3):
      http-walker: remove unused parameter from fetch_object
      http: avoid disconnecting on 404s for loose objects
      http-walker: reduce O(n) ops with doubly-linked list

 http-walker.c |  55 ++++++++++----------
 http.c        |  16 +++++-
 list.h        | 164 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 204 insertions(+), 31 deletions(-)
 create mode 100644 list.h

next             reply	other threads:[~2016-07-11 20:51 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-07-11 20:51 Eric Wong [this message]
2016-07-11 20:51 ` [PATCH 1/3] http-walker: remove unused parameter from fetch_object Eric Wong
2016-07-11 20:51 ` [PATCH 2/3] http: avoid disconnecting on 404s for loose objects Eric Wong
2016-07-11 20:51 ` [PATCH 3/3] http-walker: reduce O(n) ops with doubly-linked list Eric Wong
2016-07-11 21:02 ` [REJECT 4/3] http-walker: use hashmap to reduce list scan Eric Wong
2016-07-24 10:11 ` [RFC 0/3] dumb HTTP transport speedups Jakub Narębski
2016-07-24 11:20   ` Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20160711205131.1291-1-e@80x24.org \
    --to=e@80x24$(echo .)org \
    --cc=git@vger$(echo .)kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox