From: Johan Herland <johan@herland•net>
To: Shawn Pearce <spearce@spearce•org>
Cc: git@vger•kernel.org, Junio C Hamano <gitster@pobox•com>
Subject: Re: [PATCHv3 3/9] pack-objects: Allow --max-pack-size to be used together with --stdout
Date: Mon, 16 May 2011 03:39:25 +0200 [thread overview]
Message-ID: <201105160339.25737.johan@herland.net> (raw)
In-Reply-To: <BANLkTingc375Kzm-TBS4aQ_912pg7G03cQ@mail.gmail.com>
On Monday 16 May 2011, Shawn Pearce wrote:
> On Sun, May 15, 2011 at 14:37, Johan Herland <johan@herland•net> wrote:
> > Currently we refuse combining --max-pack-size with --stdout since
> > there's no way to make multiple packs when the pack is written to
> > stdout. However, we want to be able to limit the maximum size of the
> > pack created by --stdout (and abort pack-objects if we are unable to
> > meet that limit).
> >
> > Therefore, when used together with --stdout, we reinterpret
> > --max-pack-size to indicate the maximum pack size which - if exceeded
> > - will cause pack-objects to abort with an error message.
>
> ...
>
> > if (pack_to_stdout) {
> > + if (nr_written != nr_remaining)
> > + die("unable to make pack within the pack size"
> > + " limit (%lu bytes)", pack_size_limit);
>
> I think this is too late. We have already output a bunch of data, up
> to the size limit at this point. If the size limit is non-trivial
> (e.g. 5 MB) we have already sent most of that to the remote side, and
> its already written some of that out to disk.
>
> I'd like this to be a soft limit derived from the reused object sizes.
> When planning the pack by looking at where we will reuse an object
> from, sum those sizes. If the sum of these sizes would break this
> limit, then we abort before even writing the pack header out.
I agree, but it's currently late Sunday (early Monday), and after
looking at this for a while, I'm no longer thinking straight.
If someone that groks the pack-objects internal could help out, I'd be
really grateful. AFAICS, we need to drill into prepare_pack() to find
the details needed to estimate the total pack size, but I don't know
exactly which data structure(s) holds the data needed. We probably need
to accumulate a pack size estimate in find_deltas(), and then sum those
across the threads, before we finally compare the total estimate to
pack_size_limit prior to calling write_pack_file().
...Johan
--
Johan Herland, <johan@herland•net>
www.herland.net
next prev parent reply other threads:[~2011-05-16 1:39 UTC|newest]
Thread overview: 53+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-05-13 16:54 [PATCH 2/2] receive-pack: Add receive.denyObjectLimit to refuse push with too many objects Johan Herland
2011-05-13 17:09 ` Junio C Hamano
2011-05-14 1:43 ` Johan Herland
2011-05-14 2:03 ` [PATCHv2 2/2] receive-pack: Add receive.objectCountLimit " Johan Herland
2011-05-14 2:30 ` Shawn Pearce
2011-05-14 13:17 ` Johan Herland
2011-05-14 22:17 ` Shawn Pearce
2011-05-15 17:42 ` Johan Herland
2011-05-15 21:37 ` [PATCHv3 0/9] Push limits Johan Herland
2011-05-15 21:37 ` [PATCHv3 1/9] Update technical docs to reflect side-band-64k capability in receive-pack Johan Herland
2011-05-15 21:37 ` [PATCHv3 2/9] send-pack: Attempt to retrieve remote status even if pack-objects fails Johan Herland
2011-05-16 4:07 ` Jeff King
2011-05-16 6:13 ` Jeff King
2011-05-16 6:39 ` Jeff King
2011-05-16 6:46 ` [PATCH 1/3] connect: treat generic proxy processes like ssh processes Jeff King
2011-05-16 19:57 ` Johannes Sixt
2011-05-16 23:12 ` Junio C Hamano
2011-05-17 5:54 ` Jeff King
2011-05-17 20:14 ` Johannes Sixt
2011-05-18 8:57 ` Jeff King
2011-05-16 6:52 ` [PATCH 2/3] connect: let callers know if connection is a socket Jeff King
2011-05-16 6:52 ` [PATCH 3/3] send-pack: avoid deadlock on git:// push with failed pack-objects Jeff King
2011-05-16 20:02 ` Johannes Sixt
2011-05-17 5:56 ` Jeff King
2011-05-18 20:24 ` [PATCH] Windows: add a wrapper for the shutdown() system call Johannes Sixt
2011-05-15 21:37 ` [PATCHv3 3/9] pack-objects: Allow --max-pack-size to be used together with --stdout Johan Herland
2011-05-15 22:06 ` Shawn Pearce
2011-05-16 1:39 ` Johan Herland [this message]
2011-05-16 6:12 ` Junio C Hamano
2011-05-16 9:27 ` Johan Herland
2011-05-15 21:37 ` [PATCHv3 4/9] pack-objects: Teach new option --max-object-count, similar to --max-pack-size Johan Herland
2011-05-15 22:07 ` Shawn Pearce
2011-05-15 22:31 ` Johan Herland
2011-05-15 23:48 ` Shawn Pearce
2011-05-16 6:25 ` Junio C Hamano
2011-05-16 9:49 ` Johan Herland
2011-05-15 21:37 ` [PATCHv3 5/9] pack-objects: Teach new option --max-commit-count, limiting #commits in pack Johan Herland
2011-05-15 21:37 ` [PATCHv3 6/9] receive-pack: Prepare for addition of the new 'limit-*' family of capabilities Johan Herland
2011-05-16 6:50 ` Junio C Hamano
2011-05-16 9:53 ` Johan Herland
2011-05-16 22:02 ` Sverre Rabbelier
2011-05-16 22:07 ` Junio C Hamano
2011-05-16 22:09 ` Sverre Rabbelier
2011-05-16 22:12 ` Junio C Hamano
2011-05-16 22:16 ` Sverre Rabbelier
2011-05-15 21:37 ` [PATCHv3 7/9] send-pack/receive-pack: Allow server to refuse pushes with too many objects Johan Herland
2011-05-15 21:37 ` [PATCHv3 8/9] send-pack/receive-pack: Allow server to refuse pushing too large packs Johan Herland
2011-05-15 21:37 ` [PATCHv3 9/9] send-pack/receive-pack: Allow server to refuse pushes with too many commits Johan Herland
2011-05-15 21:52 ` [PATCHv3 0/9] Push limits Ævar Arnfjörð Bjarmason
2011-05-14 17:50 ` [PATCHv2 2/2] receive-pack: Add receive.objectCountLimit to refuse push with too many objects Junio C Hamano
2011-05-14 22:27 ` Shawn Pearce
2011-05-13 18:20 ` [PATCH 2/2] receive-pack: Add receive.denyObjectLimit " Johannes Sixt
2011-05-14 1:49 ` Johan Herland
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=201105160339.25737.johan@herland.net \
--to=johan@herland$(echo .)net \
--cc=git@vger$(echo .)kernel.org \
--cc=gitster@pobox$(echo .)com \
--cc=spearce@spearce$(echo .)org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox