* git repository size vs. subversion repository size @ 2008-04-04 22:02 Sean Brown 2008-04-04 22:17 ` Björn Steinbrink 2008-04-05 3:11 ` Shawn O. Pearce 0 siblings, 2 replies; 11+ messages in thread From: Sean Brown @ 2008-04-04 22:02 UTC (permalink / raw) To: git Last night I decided to see what storage size differences I might see between an svn repo and a git one. So I imported a highly used subversion repository into git and was shocked to see how huge the git version was. I used a repo that has a lot of branches and tagged releases just to make sure importing into git would in fact keep all of the history. It did keep the history, but the total disk usage was very different: $subversionbox # du -hs ./my_sample_website/ 67M ./my_sample_website $localhost # du -hs ./git-samplesite/ 3.6GB ./git-samplesite/ Here are the steps I took (locally): mkdir git-samplesite-tmp cd git-samplesite-tmp git-svn init http://subversion.myco.com/my_sample_website --no-metadata git config svn.authorsfile ~/Desktop/users.txt # mapped svn users to git users git-svn fetch git clone git-samplesite-tmp git-samplesite I did this based on reading the documents in the git wiki, so I assumed they were "best practice." Did I do something wrong? If this is a normal amount of storage need increase, we'd likely not move to git based on the need for new hardware alone. Any help would be appreciated. Sean ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: git repository size vs. subversion repository size 2008-04-04 22:02 git repository size vs. subversion repository size Sean Brown @ 2008-04-04 22:17 ` Björn Steinbrink 2008-04-04 23:49 ` Stephen Bannasch 2008-04-05 2:27 ` Sean Brown 2008-04-05 3:11 ` Shawn O. Pearce 1 sibling, 2 replies; 11+ messages in thread From: Björn Steinbrink @ 2008-04-04 22:17 UTC (permalink / raw) To: Sean Brown; +Cc: git On 2008.04.04 18:02:56 -0400, Sean Brown wrote: > Last night I decided to see what storage size differences I might see > between an svn repo and a git one. So I imported a highly used > subversion repository into git and was shocked to see how huge the git > version was. I used a repo that has a lot of branches and tagged > releases just to make sure importing into git would in fact keep all > of the history. It did keep the history, but the total disk usage was > very different: > > $subversionbox # du -hs ./my_sample_website/ > 67M ./my_sample_website > > $localhost # du -hs ./git-samplesite/ > 3.6GB ./git-samplesite/ How much of that is in the .git/svn directory? The contents of that directory are used to map git commits to svn revision and git versions before 1.5.4 had a quite space consuming file format for that. The new format is a lot better. If you want to switch completely, you can even just delete the .git/svn directory, as that's only required as long as you want to interact with the corresponding svn repository. And finally, you might want to repack to repository once after the initial import, to get a smaller repo. Something like: git repack -a -d -f --window=100 --depth=100 Björn ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: git repository size vs. subversion repository size 2008-04-04 22:17 ` Björn Steinbrink @ 2008-04-04 23:49 ` Stephen Bannasch 2008-04-05 0:01 ` Steven Walter 2008-04-05 2:27 ` Sean Brown 1 sibling, 1 reply; 11+ messages in thread From: Stephen Bannasch @ 2008-04-04 23:49 UTC (permalink / raw) To: git I'm just fooling around with git so far but I found a huge space savings after running git gc. Here are the rough numbers: svn repo on server: 1GB svn repo checked out: 2GB git svn clone after gc: 384MB That's saving the full history in git -- about 13000 revisions. Using git version 1.5.4.4. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: git repository size vs. subversion repository size 2008-04-04 23:49 ` Stephen Bannasch @ 2008-04-05 0:01 ` Steven Walter 2008-04-05 0:04 ` Stephen Bannasch ` (2 more replies) 0 siblings, 3 replies; 11+ messages in thread From: Steven Walter @ 2008-04-05 0:01 UTC (permalink / raw) To: Stephen Bannasch; +Cc: git On Fri, Apr 04, 2008 at 07:49:24PM -0400, Stephen Bannasch wrote: > I'm just fooling around with git so far but I found a huge space savings > after running git gc. Here are the rough numbers: > > svn repo on server: 1GB > svn repo checked out: 2GB > git svn clone after gc: 384MB > > That's saving the full history in git -- about 13000 revisions. git-gc is such an important step in importing a repository from svn. Why doesn't git-svn take this vital step automatically? -- -Steven Walter <stevenrwalter@gmail•com> Freedom is the freedom to say that 2 + 2 = 4 B2F1 0ECC E605 7321 E818 7A65 FC81 9777 DC28 9E8F ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: git repository size vs. subversion repository size 2008-04-05 0:01 ` Steven Walter @ 2008-04-05 0:04 ` Stephen Bannasch 2008-04-05 0:18 ` Björn Steinbrink 2008-04-14 15:28 ` Eric Hanchrow 2 siblings, 0 replies; 11+ messages in thread From: Stephen Bannasch @ 2008-04-05 0:04 UTC (permalink / raw) To: git >On Fri, Apr 04, 2008 at 07:49:24PM -0400, Stephen Bannasch wrote: >> I'm just fooling around with git so far but I found a huge space savings >> after running git gc. Here are the rough numbers: >> >> svn repo on server: 1GB >> svn repo checked out: 2GB >> git svn clone after gc: 384MB >> >> That's saving the full history in git -- about 13000 revisions. > >git-gc is such an important step in importing a repository from svn. >Why doesn't git-svn take this vital step automatically? I think because it is not necessary for continued productive use of git and the gc operation is expensive. On the repo above it took about 8 hours running in the background while I was working on other stuff. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: git repository size vs. subversion repository size 2008-04-05 0:01 ` Steven Walter 2008-04-05 0:04 ` Stephen Bannasch @ 2008-04-05 0:18 ` Björn Steinbrink 2008-04-14 15:28 ` Eric Hanchrow 2 siblings, 0 replies; 11+ messages in thread From: Björn Steinbrink @ 2008-04-05 0:18 UTC (permalink / raw) To: Steven Walter; +Cc: Stephen Bannasch, git [Stephan, please stop dropping me from Cc:, thanks] On 2008.04.04 20:01:41 -0400, Steven Walter wrote: > On Fri, Apr 04, 2008 at 07:49:24PM -0400, Stephen Bannasch wrote: > > I'm just fooling around with git so far but I found a huge space savings > > after running git gc. Here are the rough numbers: > > > > svn repo on server: 1GB > > svn repo checked out: 2GB > > git svn clone after gc: 384MB > > > > That's saving the full history in git -- about 13000 revisions. > > git-gc is such an important step in importing a repository from svn. > Why doesn't git-svn take this vital step automatically? Starting from 1.5.4 (IIRC) git-svn will repack every 1000 revisions (by default). That won't give you a reeeeally tiny pack but OTOH it won't take ages to do the repacks either. Björn ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: git repository size vs. subversion repository size 2008-04-05 0:01 ` Steven Walter 2008-04-05 0:04 ` Stephen Bannasch 2008-04-05 0:18 ` Björn Steinbrink @ 2008-04-14 15:28 ` Eric Hanchrow 2 siblings, 0 replies; 11+ messages in thread From: Eric Hanchrow @ 2008-04-14 15:28 UTC (permalink / raw) To: git >>>>> "Steven" == Steven Walter <stevenrwalter@gmail•com> writes: Steven> git-gc is such an important step in importing a repository Steven> from svn. Why doesn't git-svn take this vital step Steven> automatically? Mine did: git-svn version 1.5.5 (svn 1.3.2) git-svn init file://$HOME/svn-repos git-svn fetch ... M trunk/home/local/bin/spam/print-subjects.ss r5480 = b3edab03f5bacda1db025bd2cca769abbe007f23 (git-svn) Auto packing your repository for optimum performance. You may also run "git gc" manually. See "git help gc" for more information. Counting objects: 11182, done. Compressing objects: 100% (11021/11021), done. Writing objects: 100% (11182/11182), done. Total 11182 (delta 9378), reused 0 (delta 0) Checked out HEAD: file:///home/erich/svn-repos r5480 $ du -sh /tmp/ya/.git/ ~/svn-repos/ 23M /tmp/ya/.git/ 88M /home/erich/svn-repos/ -- The old graybeards in the Smalltalk world may not seem relevant, but if you ask them a question about ORM, they have been thinking about it for 20 years. -- Avi Bryant ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: git repository size vs. subversion repository size 2008-04-04 22:17 ` Björn Steinbrink 2008-04-04 23:49 ` Stephen Bannasch @ 2008-04-05 2:27 ` Sean Brown 2008-04-05 2:34 ` Björn Steinbrink 1 sibling, 1 reply; 11+ messages in thread From: Sean Brown @ 2008-04-05 2:27 UTC (permalink / raw) To: Björn Steinbrink; +Cc: git On Fri, Apr 4, 2008 at 6:17 PM, Björn Steinbrink <B.Steinbrink@gmx•de> wrote: > On 2008.04.04 18:02:56 -0400, Sean Brown wrote: > > Last night I decided to see what storage size differences I might see > > between an svn repo and a git one. So I imported a highly used > > subversion repository into git and was shocked to see how huge the git > > version was. I used a repo that has a lot of branches and tagged > > releases just to make sure importing into git would in fact keep all > > of the history. It did keep the history, but the total disk usage was > > very different: > > > > $subversionbox # du -hs ./my_sample_website/ > > 67M ./my_sample_website > > > > $localhost # du -hs ./git-samplesite/ > > 3.6GB ./git-samplesite/ > > How much of that is in the .git/svn directory? The contents of that > directory are used to map git commits to svn revision and git versions > before 1.5.4 had a quite space consuming file format for that. The new > format is a lot better. If you want to switch completely, you can even > just delete the .git/svn directory, as that's only required as long as > you want to interact with the corresponding svn repository. > > And finally, you might want to repack to repository once after the > initial import, to get a smaller repo. Something like: > git repack -a -d -f --window=100 --depth=100 > The svn folder (in the.git directory) was only about 4.2 MB. After running the repack (and then after that git gc as mentioned by another in this thread), it's still about 3.5 GB. git-samplesite (master)]$ du -hs ./* 2.1G ./branches 1.4G ./tags 66M ./trunk The site does have a lot of binary files (PDFs, photographs an such). I suppose we could leave all of the branches and tags in subversion and just move the trunk to git, but I was hoping to make a clean break from subversion. If anyone has any further suggestions I'd love to hear them. Sean -- Sean Brown seanmichaelbrown@gmail•com ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: git repository size vs. subversion repository size 2008-04-05 2:27 ` Sean Brown @ 2008-04-05 2:34 ` Björn Steinbrink 2008-04-13 9:57 ` Jan Hudec 0 siblings, 1 reply; 11+ messages in thread From: Björn Steinbrink @ 2008-04-05 2:34 UTC (permalink / raw) To: Sean Brown; +Cc: git On 2008.04.04 22:27:12 -0400, Sean Brown wrote: > On Fri, Apr 4, 2008 at 6:17 PM, Björn Steinbrink <B.Steinbrink@gmx•de> wrote: > > On 2008.04.04 18:02:56 -0400, Sean Brown wrote: > > > Last night I decided to see what storage size differences I might see > > > between an svn repo and a git one. So I imported a highly used > > > subversion repository into git and was shocked to see how huge the git > > > version was. I used a repo that has a lot of branches and tagged > > > releases just to make sure importing into git would in fact keep all > > > of the history. It did keep the history, but the total disk usage was > > > very different: > > > > > > $subversionbox # du -hs ./my_sample_website/ > > > 67M ./my_sample_website > > > > > > $localhost # du -hs ./git-samplesite/ > > > 3.6GB ./git-samplesite/ > > > > How much of that is in the .git/svn directory? The contents of that > > directory are used to map git commits to svn revision and git versions > > before 1.5.4 had a quite space consuming file format for that. The new > > format is a lot better. If you want to switch completely, you can even > > just delete the .git/svn directory, as that's only required as long as > > you want to interact with the corresponding svn repository. > > > > And finally, you might want to repack to repository once after the > > initial import, to get a smaller repo. Something like: > > git repack -a -d -f --window=100 --depth=100 > > > > The svn folder (in the.git directory) was only about 4.2 MB. After > running the repack (and then after that git gc as mentioned by another > in this thread), it's still about 3.5 GB. > > git-samplesite (master)]$ du -hs ./* > 2.1G ./branches > 1.4G ./tags > 66M ./trunk Uhm, you forgot to use -s when doing the clone. That would have created real git branches instead of the directories... What you are counting is the size of the checked out, uncompressed files of _all_ branches and _all_ tags (and trunk). The repo size of basically what "du -sh .git" would give. Björn ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: git repository size vs. subversion repository size 2008-04-05 2:34 ` Björn Steinbrink @ 2008-04-13 9:57 ` Jan Hudec 0 siblings, 0 replies; 11+ messages in thread From: Jan Hudec @ 2008-04-13 9:57 UTC (permalink / raw) To: Björn Steinbrink; +Cc: Sean Brown, git On Sat, Apr 05, 2008 at 04:34:37 +0200, Björn Steinbrink wrote: > On 2008.04.04 22:27:12 -0400, Sean Brown wrote: > > On Fri, Apr 4, 2008 at 6:17 PM, Björn Steinbrink <B.Steinbrink@gmx•de> wrote: > > > On 2008.04.04 18:02:56 -0400, Sean Brown wrote: > > > > Last night I decided to see what storage size differences I might see > > > > between an svn repo and a git one. So I imported a highly used > > > > subversion repository into git and was shocked to see how huge the git > > > > version was. I used a repo that has a lot of branches and tagged > > > > releases just to make sure importing into git would in fact keep all > > > > of the history. It did keep the history, but the total disk usage was > > > > very different: > > > > > > > > $subversionbox # du -hs ./my_sample_website/ > > > > 67M ./my_sample_website > > > > > > > > $localhost # du -hs ./git-samplesite/ > > > > 3.6GB ./git-samplesite/ > > > > > > How much of that is in the .git/svn directory? The contents of that > > > directory are used to map git commits to svn revision and git versions > > > before 1.5.4 had a quite space consuming file format for that. The new > > > format is a lot better. If you want to switch completely, you can even > > > just delete the .git/svn directory, as that's only required as long as > > > you want to interact with the corresponding svn repository. > > > > > > And finally, you might want to repack to repository once after the > > > initial import, to get a smaller repo. Something like: > > > git repack -a -d -f --window=100 --depth=100 > > > > > > > The svn folder (in the.git directory) was only about 4.2 MB. After > > running the repack (and then after that git gc as mentioned by another > > in this thread), it's still about 3.5 GB. > > > > git-samplesite (master)]$ du -hs ./* > > 2.1G ./branches > > 1.4G ./tags > > 66M ./trunk > > Uhm, you forgot to use -s when doing the clone. That would have created No, not the clone, but the git svn init. > real git branches instead of the directories... What you are counting is > the size of the checked out, uncompressed files of _all_ branches and > _all_ tags (and trunk). The repo size of basically what "du -sh .git" > would give. > > Björn > -- > To unsubscribe from this list: send the line "unsubscribe git" in > the body of a message to majordomo@vger•kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Jan 'Bulb' Hudec <bulb@ucw•cz> ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: git repository size vs. subversion repository size 2008-04-04 22:02 git repository size vs. subversion repository size Sean Brown 2008-04-04 22:17 ` Björn Steinbrink @ 2008-04-05 3:11 ` Shawn O. Pearce 1 sibling, 0 replies; 11+ messages in thread From: Shawn O. Pearce @ 2008-04-05 3:11 UTC (permalink / raw) To: Sean Brown; +Cc: git Sean Brown <seanmichaelbrown@gmail•com> wrote: > > Here are the steps I took (locally): > > mkdir git-samplesite-tmp > cd git-samplesite-tmp > git-svn init http://subversion.myco.com/my_sample_website --no-metadata > git config svn.authorsfile ~/Desktop/users.txt # mapped svn users to git users > git-svn fetch > git clone git-samplesite-tmp git-samplesite > > I did this based on reading the documents in the git wiki, so I > assumed they were "best practice." Did I do something wrong? The last command there didn't get you the most efficiently packed repository possible. More recent versions of git-clone will prefer to hardlink all of the loose objects and packs from the source to the destination, so the clone can occur more quickly when they are on the same filesystem. Really what you want to do here is repack the cloned directory (cd git-samplesite && git repack -a -d -f) and maybe include some aggressive --depth and --window options (e.g. 100/100) if you have some CPU time to burn and are reasonably certain you will be keeping the result. You only have to spend that CPU time once when converting from SVN, and all future clones from this one will benefit. But your really major disk usage was due to what someone else pointed out, which was missing the "-s" flag to git-svn. So the Git working directory was huge, as we created working files for every single branch and every single tag. Ouch. -- Shawn. ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2008-04-14 15:39 UTC | newest] Thread overview: 11+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2008-04-04 22:02 git repository size vs. subversion repository size Sean Brown 2008-04-04 22:17 ` Björn Steinbrink 2008-04-04 23:49 ` Stephen Bannasch 2008-04-05 0:01 ` Steven Walter 2008-04-05 0:04 ` Stephen Bannasch 2008-04-05 0:18 ` Björn Steinbrink 2008-04-14 15:28 ` Eric Hanchrow 2008-04-05 2:27 ` Sean Brown 2008-04-05 2:34 ` Björn Steinbrink 2008-04-13 9:57 ` Jan Hudec 2008-04-05 3:11 ` Shawn O. Pearce
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox