From: "Shawn O. Pearce" <spearce@spearce•org>
To: Sebastian Bober <sbober@servercare•de>
Cc: Sverre Rabbelier <srabbelier@gmail•com>,
Richard Hartmann <richih.mailinglist@gmail•com>,
Git List <git@vger•kernel.org>,
Avery Pennarun <apenwarr@gmail•com>,
Nicolas Pitre <nico@fluxnic•net>, Sam Vilain <sam@vilain•net>
Subject: Re: Git import of the recent full enwiki dump
Date: Fri, 16 Apr 2010 17:53:42 -0700 [thread overview]
Message-ID: <20100417005342.GA8475@spearce.org> (raw)
In-Reply-To: <20100417004852.GA32053@post.servercare.de>
Sebastian Bober <sbober@servercare•de> wrote:
> The question would be, how the commits and the trees are laid out.
> If every wiki revision shall be a git commit, then we'd need to handle
> 300M commits. And we have 19M wiki pages (that would be files). The tree
> objects would be very large and git-fast-import would crawl.
>
> Some tests with the german wikipedia have shown that importing the blobs
> is doable on normal hardware. Getting the trees and commits into git
> was not possible up to now, as fast-import was just to slow (and getting
> slower after 1M commits).
Well, to be fair to fast-import, its tree handling code is linear
scan based, because that's how any other part of Git handles trees.
If you just toss all 19M wiki pages into a single top level tree,
that's going to take a very long time to locate the wiki page
talking about Zoos.
> I had the idea of having an importer that would just handle this special
> case (1 file change per commit), but didn't get around to try that yet.
Really, fast-import should be able to handle this well, assuming you
aren't just tossing all 19M files into a single massive directory
and hoping for the best. Because *any* program working on that
sort of layout will need to spit out the 19M entry tree object on
each and every commit, just so it can compute the SHA-1 checksum
to get the tree name for the commit.
--
Shawn.
next prev parent reply other threads:[~2010-04-17 1:02 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-04-16 23:47 Git import of the recent full enwiki dump Richard Hartmann
2010-04-17 0:19 ` Sverre Rabbelier
2010-04-17 0:48 ` Sebastian Bober
2010-04-17 0:53 ` Shawn O. Pearce [this message]
2010-04-17 1:01 ` Sebastian Bober
2010-04-17 1:44 ` [spf:guess] " Sam Vilain
2010-04-17 1:58 ` Sebastian Bober
2010-04-17 3:34 ` [spf:guess] " Sam Vilain
2010-04-17 7:48 ` Sebastian Bober
2010-04-17 1:10 ` Richard Hartmann
2010-04-17 1:18 ` Shawn O. Pearce
2010-04-17 1:25 ` Sebastian Bober
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20100417005342.GA8475@spearce.org \
--to=spearce@spearce$(echo .)org \
--cc=apenwarr@gmail$(echo .)com \
--cc=git@vger$(echo .)kernel.org \
--cc=nico@fluxnic$(echo .)net \
--cc=richih.mailinglist@gmail$(echo .)com \
--cc=sam@vilain$(echo .)net \
--cc=sbober@servercare$(echo .)de \
--cc=srabbelier@gmail$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox