From: Junio C Hamano <gitster@pobox•com>
To: Nicolas Pitre <nico@fluxnic•net>
Cc: git@vger•kernel.org
Subject: Re: [PATCH 14/23] pack v4: object data copy
Date: Tue, 27 Aug 2013 08:53:44 -0700 [thread overview]
Message-ID: <xmqqli3ndt9z.fsf@gitster.dls.corp.google.com> (raw)
In-Reply-To: <1377577567-27655-15-git-send-email-nico@fluxnic.net> (Nicolas Pitre's message of "Tue, 27 Aug 2013 00:25:58 -0400")
Nicolas Pitre <nico@fluxnic•net> writes:
> Blob and tag objects have no particular changes except for their object
> header.
>
> Delta objects are also copied as is, except for their delta base reference
> which is converted to the new way as used elsewhere in pack v4 encoding
> i.e. an index into the SHA1 table or a literal SHA1 prefixed by 0 if not
> found in the table (see add_sha1_ref). This is true for both REF_DELTA
> as well as OFS_DELTA.
>
> Object payload is validated against the recorded CRC32 in the source
> pack index file when possible.
>
> Signed-off-by: Nicolas Pitre <nico@fluxnic•net>
> ---
The title somewhat confused me until I realized that this series is
building a program that would convert existing data from a single
pack into packv4 format, not a "pack-objects --pack-verison=4".
> packv4-create.c | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 66 insertions(+)
>
> diff --git a/packv4-create.c b/packv4-create.c
> index 6e0bb1d..a6dc818 100644
> --- a/packv4-create.c
> +++ b/packv4-create.c
> @@ -12,6 +12,7 @@
> #include "object.h"
> #include "tree-walk.h"
> #include "pack.h"
> +#include "pack-revindex.h"
>
>
> static int pack_compression_level = Z_DEFAULT_COMPRESSION;
> @@ -673,6 +674,71 @@ static unsigned int write_object_header(struct sha1file *f, enum object_type typ
> return end - buf;
> }
>
> +static unsigned long copy_object_data(struct sha1file *f, struct packed_git *p,
> + off_t offset)
> +{
> + struct pack_window *w_curs = NULL;
> + struct revindex_entry *revidx;
> + enum object_type type;
> + unsigned long avail, size, datalen, written;
> + int hdrlen, idx_nr;
> + unsigned char *src, *end, buf[24];
> +
> + revidx = find_pack_revindex(p, offset);
> + idx_nr = revidx->nr;
> + datalen = revidx[1].offset - offset;
> +
> + src = use_pack(p, &w_curs, offset, &avail);
> + hdrlen = unpack_object_header_buffer(src, avail, &type, &size);
> +
> + written = write_object_header(f, type, size);
> +
> + if (type == OBJ_OFS_DELTA) {
> + unsigned char c = src[hdrlen++];
> + off_t base_offset = c & 127;
> + while (c & 128) {
> + base_offset += 1;
> + if (!base_offset || MSB(base_offset, 7))
> + die("delta offset overflow");
> + c = src[hdrlen++];
> + base_offset = (base_offset << 7) + (c & 127);
> + }
> + base_offset = offset - base_offset;
> + if (base_offset <= 0 || base_offset >= offset)
> + die("delta offset out of bound");
> + revidx = find_pack_revindex(p, base_offset);
> + end = add_sha1_ref(buf, nth_packed_object_sha1(p, revidx->nr));
> + sha1write(f, buf, end - buf);
> + written += end - buf;
> + } else if (type == OBJ_REF_DELTA) {
> + end = add_sha1_ref(buf, src + hdrlen);
> + hdrlen += 20;
> + sha1write(f, buf, end - buf);
> + written += end - buf;
> + }
> +
> + if (p->index_version > 1 &&
> + check_pack_crc(p, &w_curs, offset, datalen, idx_nr))
> + die("bad CRC for object at offset %"PRIuMAX" in %s",
> + (uintmax_t)offset, p->pack_name);
> +
> + offset += hdrlen;
> + datalen -= hdrlen;
> +
> + while (datalen) {
> + src = use_pack(p, &w_curs, offset, &avail);
> + if (avail > datalen)
> + avail = datalen;
> + sha1write(f, src, avail);
> + written += avail;
> + offset += avail;
> + datalen -= avail;
> + }
> + unuse_pack(&w_curs);
> +
> + return written;
> +}
> +
> static struct packed_git *open_pack(const char *path)
> {
> char arg[PATH_MAX];
next prev parent reply other threads:[~2013-08-27 15:53 UTC|newest]
Thread overview: 83+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-08-27 4:25 [PATCH 00/23] Preliminary pack v4 support Nicolas Pitre
2013-08-27 4:25 ` [PATCH 01/23] pack v4: initial pack dictionary structure and code Nicolas Pitre
2013-08-27 15:08 ` Junio C Hamano
2013-08-27 16:13 ` Nicolas Pitre
2013-08-27 4:25 ` [PATCH 02/23] export packed_object_info() Nicolas Pitre
2013-08-27 4:25 ` [PATCH 03/23] pack v4: scan tree objects Nicolas Pitre
2013-08-27 4:25 ` [PATCH 04/23] pack v4: add tree entry mode support to dictionary entries Nicolas Pitre
2013-08-27 4:25 ` [PATCH 05/23] pack v4: add commit object parsing Nicolas Pitre
2013-08-27 15:26 ` Junio C Hamano
2013-08-27 16:47 ` Nicolas Pitre
2013-08-27 17:42 ` Junio C Hamano
2013-08-27 4:25 ` [PATCH 06/23] pack v4: split the object list and dictionary creation Nicolas Pitre
2013-08-27 4:25 ` [PATCH 07/23] pack v4: move to struct pack_idx_entry and get rid of our own struct idx_entry Nicolas Pitre
2013-08-27 4:25 ` [PATCH 08/23] pack v4: basic references encoding Nicolas Pitre
2013-08-27 15:29 ` Junio C Hamano
2013-08-27 15:53 ` Nicolas Pitre
2013-08-27 4:25 ` [PATCH 09/23] pack v4: commit object encoding Nicolas Pitre
2013-08-27 15:39 ` Junio C Hamano
2013-08-27 16:50 ` Nicolas Pitre
2013-08-27 19:59 ` Nicolas Pitre
2013-08-27 20:15 ` Junio C Hamano
2013-08-27 21:43 ` Nicolas Pitre
2013-09-02 20:48 ` Duy Nguyen
2013-09-03 6:30 ` Nicolas Pitre
2013-09-03 7:41 ` Duy Nguyen
2013-09-05 3:50 ` Nicolas Pitre
2013-08-27 4:25 ` [PATCH 10/23] pack v4: tree " Nicolas Pitre
2013-08-27 15:44 ` Junio C Hamano
2013-08-27 16:52 ` Nicolas Pitre
2013-08-27 4:25 ` [PATCH 11/23] pack v4: dictionary table output Nicolas Pitre
2013-08-27 4:25 ` [PATCH 12/23] pack v4: creation code Nicolas Pitre
2013-08-27 15:48 ` Junio C Hamano
2013-08-27 16:59 ` Nicolas Pitre
2013-08-27 4:25 ` [PATCH 13/23] pack v4: object headers Nicolas Pitre
2013-08-27 4:25 ` [PATCH 14/23] pack v4: object data copy Nicolas Pitre
2013-08-27 15:53 ` Junio C Hamano [this message]
2013-08-27 18:24 ` Nicolas Pitre
2013-08-27 4:25 ` [PATCH 15/23] pack v4: object writing Nicolas Pitre
2013-08-27 4:26 ` [PATCH 16/23] pack v4: tree object delta encoding Nicolas Pitre
2013-08-27 4:26 ` [PATCH 17/23] pack v4: load delta candidate for encoding tree objects Nicolas Pitre
2013-08-27 4:26 ` [PATCH 18/23] pack v4: honor pack.compression config option Nicolas Pitre
2013-08-27 4:26 ` [PATCH 19/23] pack v4: relax commit parsing a bit Nicolas Pitre
2013-08-27 4:26 ` [PATCH 20/23] pack index v3 Nicolas Pitre
2013-08-27 4:26 ` [PATCH 21/23] pack v4: normalize pack name to properly generate the pack index file name Nicolas Pitre
2013-08-27 4:26 ` [PATCH 22/23] pack v4: add progress display Nicolas Pitre
2013-08-27 4:26 ` [PATCH 23/23] initial pack index v3 support on the read side Nicolas Pitre
2013-08-31 11:45 ` Duy Nguyen
2013-09-03 6:09 ` Nicolas Pitre
2013-09-03 7:34 ` Duy Nguyen
2013-08-27 11:17 ` [PATCH] Document pack v4 format Nguyễn Thái Ngọc Duy
2013-08-27 18:25 ` Junio C Hamano
2013-08-27 18:53 ` Nicolas Pitre
2013-08-31 2:49 ` [PATCH v2] " Nguyễn Thái Ngọc Duy
2013-09-03 6:00 ` Nicolas Pitre
2013-09-03 6:46 ` Nicolas Pitre
2013-09-03 11:49 ` Duy Nguyen
2013-09-03 14:54 ` Duy Nguyen
2013-09-05 4:12 ` Nicolas Pitre
2013-09-05 4:19 ` Duy Nguyen
2013-09-05 4:40 ` Nicolas Pitre
2013-09-05 5:04 ` Duy Nguyen
2013-09-05 5:39 ` Nicolas Pitre
2013-09-05 16:52 ` Duy Nguyen
2013-09-05 17:14 ` Nicolas Pitre
2013-09-05 20:26 ` Junio C Hamano
2013-09-05 21:04 ` Nicolas Pitre
2013-09-06 4:18 ` Duy Nguyen
2013-09-06 13:19 ` Nicolas Pitre
2013-09-06 2:14 ` [PATCH v3] " Nguyễn Thái Ngọc Duy
2013-09-06 3:23 ` Nicolas Pitre
2013-09-06 9:48 ` Duy Nguyen
2013-09-06 13:25 ` Nicolas Pitre
2013-09-06 13:44 ` Duy Nguyen
2013-09-06 16:44 ` Nicolas Pitre
2013-09-07 4:57 ` Nicolas Pitre
2013-09-07 4:52 ` Nicolas Pitre
2013-09-07 8:05 ` Duy Nguyen
2013-08-27 15:03 ` [PATCH 00/23] Preliminary pack v4 support Junio C Hamano
2013-08-27 15:59 ` Nicolas Pitre
2013-08-27 16:44 ` Junio C Hamano
2013-08-28 2:30 ` Duy Nguyen
2013-08-28 2:58 ` Nicolas Pitre
2013-08-28 3:06 ` Duy Nguyen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=xmqqli3ndt9z.fsf@gitster.dls.corp.google.com \
--to=gitster@pobox$(echo .)com \
--cc=git@vger$(echo .)kernel.org \
--cc=nico@fluxnic$(echo .)net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox