From: "brian m. carlson" <sandals@crustytoothpaste•net>
To: Lars Schneider <larsxschneider@gmail•com>
Cc: git <git@vger•kernel.org>, Jeff King <peff@peff•net>,
Taylor Blau <me@ttaylorr•com>
Subject: Re: Import/Export as a fast way to purge files from Git?
Date: Sun, 23 Sep 2018 15:53:38 +0000 [thread overview]
Message-ID: <20180923155338.GF432229@genre.crustytoothpaste.net> (raw)
In-Reply-To: <F65AF000-7AE0-44C8-81C8-E58D6769FAA3@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1081 bytes --]
On Sun, Sep 23, 2018 at 03:04:58PM +0200, Lars Schneider wrote:
> Hi,
>
> I recently had to purge files from large Git repos (many files, many commits).
> The usual recommendation is to use `git filter-branch --index-filter` to purge
> files. However, this is *very* slow for large repos (e.g. it takes 45min to
> remove the `builtin` directory from git core). I realized that I can remove
> files *way* faster by exporting the repo, removing the file references,
> and then importing the repo (see Perl script below, it takes ~30sec to remove
> the `builtin` directory from git core). Do you see any problem with this
> approach?
I don't know of any problems with this approach. I didn't audit your
specific Perl script for any issues, though.
I suspect you're gaining speed mostly because you're running three
processes total instead of at least one process (sh) per commit. So I
don't think there's anything that Git can do to make this faster on our
end without a redesign.
--
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 868 bytes --]
next prev parent reply other threads:[~2018-09-23 15:54 UTC|newest]
Thread overview: 90+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-09-23 13:04 Import/Export as a fast way to purge files from Git? Lars Schneider
2018-09-23 14:55 ` Eric Sunshine
2018-09-23 15:58 ` Lars Schneider
2018-09-23 15:53 ` brian m. carlson [this message]
2018-09-23 17:04 ` Jeff King
2018-09-24 17:24 ` Elijah Newren
2018-10-31 19:15 ` Lars Schneider
2018-11-01 7:12 ` Elijah Newren
2018-11-11 6:23 ` [PATCH 00/10] fast export and import fixes and features Elijah Newren
2018-11-11 6:23 ` [PATCH 01/10] git-fast-import.txt: fix documentation for --quiet option Elijah Newren
2018-11-11 6:33 ` Jeff King
2018-11-11 6:23 ` [PATCH 02/10] git-fast-export.txt: clarify misleading documentation about rev-list args Elijah Newren
2018-11-11 6:36 ` Jeff King
2018-11-11 7:17 ` Elijah Newren
2018-11-13 23:25 ` Elijah Newren
2018-11-13 23:39 ` Jonathan Nieder
2018-11-14 0:02 ` Elijah Newren
2018-11-11 6:23 ` [PATCH 03/10] fast-export: use value from correct enum Elijah Newren
2018-11-11 6:36 ` Jeff King
2018-11-11 20:10 ` Ævar Arnfjörð Bjarmason
2018-11-12 9:12 ` Ævar Arnfjörð Bjarmason
2018-11-12 11:31 ` Jeff King
2018-11-11 6:23 ` [PATCH 04/10] fast-export: avoid dying when filtering by paths and old tags exist Elijah Newren
2018-11-11 6:44 ` Jeff King
2018-11-11 7:38 ` Elijah Newren
2018-11-12 12:32 ` Jeff King
2018-11-12 22:50 ` brian m. carlson
2018-11-13 14:38 ` Jeff King
2018-11-11 6:23 ` [PATCH 05/10] fast-export: move commit rewriting logic into a function for reuse Elijah Newren
2018-11-11 6:47 ` Jeff King
2018-11-11 6:23 ` [PATCH 06/10] fast-export: when using paths, avoid corrupt stream with non-existent mark Elijah Newren
2018-11-11 6:53 ` Jeff King
2018-11-11 8:01 ` Elijah Newren
2018-11-12 12:45 ` Jeff King
2018-11-12 15:36 ` Elijah Newren
2018-11-11 6:23 ` [PATCH 07/10] fast-export: ensure we export requested refs Elijah Newren
2018-11-11 7:02 ` Jeff King
2018-11-11 8:20 ` Elijah Newren
2018-11-11 6:23 ` [PATCH 08/10] fast-export: add --reference-excluded-parents option Elijah Newren
2018-11-11 7:11 ` Jeff King
2018-11-11 6:23 ` [PATCH 09/10] fast-export: add a --show-original-ids option to show original names Elijah Newren
2018-11-11 7:20 ` Jeff King
2018-11-11 8:32 ` Elijah Newren
2018-11-12 12:53 ` Jeff King
2018-11-12 15:46 ` Elijah Newren
2018-11-12 16:31 ` Jeff King
2018-11-11 6:23 ` [PATCH 10/10] fast-export: add --always-show-modify-after-rename Elijah Newren
2018-11-11 7:23 ` Jeff King
2018-11-11 8:42 ` Elijah Newren
2018-11-12 12:58 ` Jeff King
2018-11-12 18:08 ` Elijah Newren
2018-11-13 14:45 ` Jeff King
2018-11-13 17:10 ` Elijah Newren
2018-11-14 7:14 ` Jeff King
2018-11-11 7:27 ` [PATCH 00/10] fast export and import fixes and features Jeff King
2018-11-11 8:44 ` Elijah Newren
2018-11-12 13:00 ` Jeff King
2018-11-14 0:25 ` [PATCH v2 00/11] " Elijah Newren
2018-11-14 0:25 ` [PATCH v2 01/11] git-fast-import.txt: fix documentation for --quiet option Elijah Newren
2018-11-14 0:25 ` [PATCH v2 02/11] git-fast-export.txt: clarify misleading documentation about rev-list args Elijah Newren
2018-11-14 0:25 ` [PATCH v2 03/11] fast-export: use value from correct enum Elijah Newren
2018-11-14 0:25 ` [PATCH v2 04/11] fast-export: avoid dying when filtering by paths and old tags exist Elijah Newren
2018-11-14 19:17 ` SZEDER Gábor
2018-11-14 23:13 ` Elijah Newren
2018-11-14 0:25 ` [PATCH v2 05/11] fast-export: move commit rewriting logic into a function for reuse Elijah Newren
2018-11-14 0:25 ` [PATCH v2 06/11] fast-export: when using paths, avoid corrupt stream with non-existent mark Elijah Newren
2018-11-14 0:25 ` [PATCH v2 07/11] fast-export: ensure we export requested refs Elijah Newren
2018-11-14 0:25 ` [PATCH v2 08/11] fast-export: add --reference-excluded-parents option Elijah Newren
2018-11-14 19:27 ` SZEDER Gábor
2018-11-14 23:16 ` Elijah Newren
2018-11-14 0:25 ` [PATCH v2 09/11] fast-import: remove unmaintained duplicate documentation Elijah Newren
2018-11-14 0:25 ` [PATCH v2 10/11] fast-export: add a --show-original-ids option to show original names Elijah Newren
2018-11-14 0:26 ` [PATCH v2 11/11] fast-export: add --always-show-modify-after-rename Elijah Newren
2018-11-14 7:25 ` [PATCH v2 00/11] fast export and import fixes and features Jeff King
2018-11-16 7:59 ` [PATCH v3 " Elijah Newren
2018-11-16 7:59 ` [PATCH v3 01/11] fast-export: convert sha1 to oid Elijah Newren
2018-11-16 7:59 ` [PATCH v3 02/11] git-fast-import.txt: fix documentation for --quiet option Elijah Newren
2018-11-16 7:59 ` [PATCH v3 03/11] git-fast-export.txt: clarify misleading documentation about rev-list args Elijah Newren
2018-11-16 7:59 ` [PATCH v3 04/11] fast-export: use value from correct enum Elijah Newren
2018-11-16 7:59 ` [PATCH v3 05/11] fast-export: avoid dying when filtering by paths and old tags exist Elijah Newren
2018-11-16 7:59 ` [PATCH v3 06/11] fast-export: move commit rewriting logic into a function for reuse Elijah Newren
2018-11-16 7:59 ` [PATCH v3 07/11] fast-export: when using paths, avoid corrupt stream with non-existent mark Elijah Newren
2018-11-16 7:59 ` [PATCH v3 08/11] fast-export: ensure we export requested refs Elijah Newren
2018-11-16 7:59 ` [PATCH v3 09/11] fast-export: add --reference-excluded-parents option Elijah Newren
2018-11-16 7:59 ` [PATCH v3 10/11] fast-import: remove unmaintained duplicate documentation Elijah Newren
2018-11-16 7:59 ` [PATCH v3 11/11] fast-export: add a --show-original-ids option to show original names Elijah Newren
2018-11-16 12:29 ` SZEDER Gábor
2018-11-16 8:50 ` [PATCH v3 00/11] fast export and import fixes and features Jeff King
2018-11-12 9:17 ` Import/Export as a fast way to purge files from Git? Ævar Arnfjörð Bjarmason
2018-11-12 15:34 ` Elijah Newren
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180923155338.GF432229@genre.crustytoothpaste.net \
--to=sandals@crustytoothpaste$(echo .)net \
--cc=git@vger$(echo .)kernel.org \
--cc=larsxschneider@gmail$(echo .)com \
--cc=me@ttaylorr$(echo .)com \
--cc=peff@peff$(echo .)net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox