From: Taylor Blau <me@ttaylorr•com>
To: git@vger•kernel.org
Cc: Jeff King <peff@peff•net>, Elijah Newren <newren@gmail•com>,
Patrick Steinhardt <ps@pks•im>,
Junio C Hamano <gitster@pobox•com>
Subject: [RFC PATCH 04/14] midx: introduce `--checksum-only` for incremental MIDX writes
Date: Tue, 24 Feb 2026 19:21:10 -0500 [thread overview]
Message-ID: <2d31ea3907ff1965302acb089a594746c8dfb149.1771978829.git.me@ttaylorr.com> (raw)
In-Reply-To: <cover.1771978829.git.me@ttaylorr.com>
When writing an incremental MIDX layer, the MIDX machinery writes the
new layer into the multi-pack-index.d directory and then updates the
multi-pack-index-chain file to include the freshly written layer.
Future callers however may not wish to immediately update the MIDX chain
itself, preferring instead to write out new layer(s) itself before
atomically updating the chain. Concretely, the new incremental
MIDX-based repacking strategy will want to do exactly this (that is,
assemble the new MIDX chain itself before writing a new chain file and
atomically linking it into place).
Introduce a `--checksum-only` flag that:
* writes the new MIDX layer into the multi-pack-index.d directory
* prints its checksum
* does not update the multi-pack-index-chain file.
The MIDX chain file (and thus, the lock protecting it) remain untouched,
allowing callers to assemble the chain themselves. This flag requires
`--incremental`, since the notion of a separate layer only makes sense
for incremental MIDXs.
Signed-off-by: Taylor Blau <me@ttaylorr•com>
---
Documentation/git-multi-pack-index.adoc | 4 +--
builtin/multi-pack-index.c | 28 ++++++++++++++++--
midx-write.c | 38 ++++++++++++++++---------
midx.h | 1 +
t/t5334-incremental-multi-pack-index.sh | 17 +++++++++++
t/t5335-compact-multi-pack-index.sh | 34 ++++++++++++++++++++++
6 files changed, 105 insertions(+), 17 deletions(-)
diff --git a/Documentation/git-multi-pack-index.adoc b/Documentation/git-multi-pack-index.adoc
index 61256830141..657e0639f6a 100644
--- a/Documentation/git-multi-pack-index.adoc
+++ b/Documentation/git-multi-pack-index.adoc
@@ -11,9 +11,9 @@ SYNOPSIS
[verse]
'git multi-pack-index' [<options>] write [--preferred-pack=<pack>]
[--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]
- [--refs-snapshot=<path>]
+ [--refs-snapshot=<path>] [--[no-]checksum-only]
'git multi-pack-index' [<options>] compact [--[no-]incremental]
- [--[no-]bitmap] <from> <to>
+ [--[no-]bitmap] [--[no-]checksum-only] <from> <to>
'git multi-pack-index' [<options>] verify
'git multi-pack-index' [<options>] expire
'git multi-pack-index' [<options>] repack [--batch-size=<size>]
diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c
index 2f24c113c8f..ee1ddf1386f 100644
--- a/builtin/multi-pack-index.c
+++ b/builtin/multi-pack-index.c
@@ -15,11 +15,11 @@
#define BUILTIN_MIDX_WRITE_USAGE \
N_("git multi-pack-index [<options>] write [--preferred-pack=<pack>]\n" \
" [--[no-]bitmap] [--[no-]incremental] [--[no-]stdin-packs]\n" \
- " [--refs-snapshot=<path>]")
+ " [--refs-snapshot=<path>] [--[no-]checksum-only]")
#define BUILTIN_MIDX_COMPACT_USAGE \
N_("git multi-pack-index [<options>] compact [--[no-]incremental]\n" \
- " [--[no-]bitmap] <from> <to>")
+ " [--[no-]bitmap] [--[no-]checksum-only] <from> <to>")
#define BUILTIN_MIDX_VERIFY_USAGE \
N_("git multi-pack-index [<options>] verify")
@@ -152,6 +152,9 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,
MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),
OPT_BIT(0, "incremental", &opts.flags,
N_("write a new incremental MIDX"), MIDX_WRITE_INCREMENTAL),
+ OPT_BIT(0, "checksum-only", &opts.flags,
+ N_("write a MIDX layer without updating the MIDX chain"),
+ MIDX_WRITE_CHECKSUM_ONLY),
OPT_BOOL(0, "stdin-packs", &opts.stdin_packs,
N_("write multi-pack index containing only given indexes")),
OPT_FILENAME(0, "refs-snapshot", &opts.refs_snapshot,
@@ -177,6 +180,15 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,
if (argc)
usage_with_options(builtin_multi_pack_index_write_usage,
options);
+
+ if (opts.flags & MIDX_WRITE_CHECKSUM_ONLY &&
+ !(opts.flags & MIDX_WRITE_INCREMENTAL)) {
+ error(_("cannot use %s without %s"),
+ "--checksum-only", "--incremental");
+ usage_with_options(builtin_multi_pack_index_write_usage,
+ options);
+ }
+
source = handle_object_dir_option(repo);
FREE_AND_NULL(options);
@@ -220,6 +232,9 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,
MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),
OPT_BIT(0, "incremental", &opts.flags,
N_("write a new incremental MIDX"), MIDX_WRITE_INCREMENTAL),
+ OPT_BIT(0, "checksum-only", &opts.flags,
+ N_("write a MIDX layer without updating the MIDX chain"),
+ MIDX_WRITE_CHECKSUM_ONLY),
OPT_END(),
};
@@ -238,6 +253,15 @@ static int cmd_multi_pack_index_compact(int argc, const char **argv,
if (argc != 2)
usage_with_options(builtin_multi_pack_index_compact_usage,
options);
+
+ if (opts.flags & MIDX_WRITE_CHECKSUM_ONLY &&
+ !(opts.flags & MIDX_WRITE_INCREMENTAL)) {
+ error(_("cannot use %s without %s"),
+ "--checksum-only", "--incremental");
+ usage_with_options(builtin_multi_pack_index_compact_usage,
+ options);
+ }
+
source = handle_object_dir_option(the_repository);
FREE_AND_NULL(options);
diff --git a/midx-write.c b/midx-write.c
index 7fdc4d31243..9cf085ca333 100644
--- a/midx-write.c
+++ b/midx-write.c
@@ -1598,11 +1598,14 @@ static int write_midx_internal(struct write_midx_opts *opts)
}
if (ctx.incremental) {
- struct strbuf lock_name = STRBUF_INIT;
+ if (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY)) {
+ struct strbuf lock_name = STRBUF_INIT;
- get_midx_chain_filename(opts->source, &lock_name);
- hold_lock_file_for_update(&lk, lock_name.buf, LOCK_DIE_ON_ERROR);
- strbuf_release(&lock_name);
+ get_midx_chain_filename(opts->source, &lock_name);
+ hold_lock_file_for_update(&lk, lock_name.buf,
+ LOCK_DIE_ON_ERROR);
+ strbuf_release(&lock_name);
+ }
incr = mks_tempfile_m(midx_name.buf, 0444);
if (!incr) {
@@ -1723,14 +1726,19 @@ static int write_midx_internal(struct write_midx_opts *opts)
}
strvec_init_alloc(&keep_hashes, keep_hashes_nr);
+ if (opts->flags & MIDX_WRITE_CHECKSUM_ONLY)
+ printf("%s\n", hash_to_hex_algop(midx_hash, r->hash_algo));
+
if (ctx.incremental) {
- FILE *chainf = fdopen_lock_file(&lk, "w");
struct strbuf final_midx_name = STRBUF_INIT;
struct multi_pack_index *m = ctx.base_midx;
- if (!chainf) {
- error_errno(_("unable to open multi-pack-index chain file"));
- goto cleanup;
+ if (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY)) {
+ FILE *chainf = fdopen_lock_file(&lk, "w");
+ if (!chainf) {
+ error_errno(_("unable to open multi-pack-index chain file"));
+ goto cleanup;
+ }
}
if (link_midx_to_chain(ctx.base_midx) < 0)
@@ -1791,8 +1799,10 @@ static int write_midx_internal(struct write_midx_opts *opts)
}
}
- for (uint32_t i = 0; i < keep_hashes_nr; i++)
- fprintf(get_lock_file_fp(&lk), "%s\n", keep_hashes.v[i]);
+ if (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY))
+ for (uint32_t i = 0; i < keep_hashes_nr; i++)
+ fprintf(get_lock_file_fp(&lk), "%s\n",
+ keep_hashes.v[i]);
} else {
keep_hashes.v[ctx.num_multi_pack_indexes_before] =
xstrdup(hash_to_hex_algop(midx_hash, r->hash_algo));
@@ -1802,10 +1812,12 @@ static int write_midx_internal(struct write_midx_opts *opts)
if (ctx.m || ctx.base_midx)
odb_close(ctx.repo->objects);
- if (commit_lock_file(&lk) < 0)
- die_errno(_("could not write multi-pack-index"));
+ if (!(opts->flags & MIDX_WRITE_CHECKSUM_ONLY)) {
+ if (commit_lock_file(&lk) < 0)
+ die_errno(_("could not write multi-pack-index"));
- clear_midx_files(opts->source, &keep_hashes, ctx.incremental);
+ clear_midx_files(opts->source, &keep_hashes, ctx.incremental);
+ }
result = 0;
cleanup:
diff --git a/midx.h b/midx.h
index 08f3728e520..9f1acd7ace4 100644
--- a/midx.h
+++ b/midx.h
@@ -83,6 +83,7 @@ struct multi_pack_index {
#define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4)
#define MIDX_WRITE_INCREMENTAL (1 << 5)
#define MIDX_WRITE_COMPACT (1 << 6)
+#define MIDX_WRITE_CHECKSUM_ONLY (1 << 7)
#define MIDX_EXT_REV "rev"
#define MIDX_EXT_BITMAP "bitmap"
diff --git a/t/t5334-incremental-multi-pack-index.sh b/t/t5334-incremental-multi-pack-index.sh
index d30d7253d6f..96449178c07 100755
--- a/t/t5334-incremental-multi-pack-index.sh
+++ b/t/t5334-incremental-multi-pack-index.sh
@@ -95,6 +95,23 @@ test_expect_success 'show object from second pack' '
git cat-file -p 2.2
'
+test_expect_success 'write MIDX layer with --checksum-only' '
+ test_commit checksum-only &&
+ git repack -d &&
+
+ cp "$midx_chain" "$midx_chain.bak" &&
+ layer="$(git multi-pack-index write --bitmap --incremental \
+ --checksum-only)" &&
+
+ test_cmp "$midx_chain.bak" "$midx_chain" &&
+ test_path_is_file "$midxdir/multi-pack-index-$layer.midx"
+'
+
+test_expect_success 'write non-incremental MIDX layer with --checksum-only' '
+ test_must_fail git multi-pack-index write --bitmap --checksum-only 2>err &&
+ test_grep "cannot use --checksum-only without --incremental" err
+'
+
for reuse in false single multi
do
test_expect_success "full clone (pack.allowPackReuse=$reuse)" '
diff --git a/t/t5335-compact-multi-pack-index.sh b/t/t5335-compact-multi-pack-index.sh
index 40f3844282f..55b9773568b 100755
--- a/t/t5335-compact-multi-pack-index.sh
+++ b/t/t5335-compact-multi-pack-index.sh
@@ -290,4 +290,38 @@ test_expect_success 'MIDX compaction with bitmaps (non-trivial)' '
)
'
+test_expect_success 'MIDX compaction with --checksum-only' '
+ git init midx-compact-with--checksum-only &&
+ (
+ cd midx-compact-with--checksum-only &&
+
+ write_packs A B C D &&
+
+ test_line_count = 4 $midx_chain &&
+ cp "$midx_chain" "$midx_chain".bak &&
+
+ layer="$(git multi-pack-index compact --incremental \
+ --checksum-only \
+ "$(nth_line 2 "$midx_chain")" \
+ "$(nth_line 3 "$midx_chain")")" &&
+
+ test_cmp "$midx_chain.bak" "$midx_chain" &&
+
+ # After writing the new layer, insert it into the chain
+ # manually. This is done in order to make $layer visible
+ # to the read-midx test helper below, and matches what
+ # the MIDX command would do without --checksum-only.
+ {
+ nth_line 1 "$midx_chain.bak" &&
+ echo $layer &&
+ nth_line 4 "$midx_chain.bak"
+ } >$midx_chain &&
+
+ test-tool read-midx $objdir $layer >midx.data &&
+ grep "^pack-B-.*\.idx" midx.data &&
+ grep "^pack-C-.*\.idx" midx.data
+
+ )
+'
+
test_done
--
2.53.0.185.g29bc4dff628
next prev parent reply other threads:[~2026-02-25 0:21 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-02-25 0:20 [RFC PATCH 00/14] repack: incremental MIDX/bitmap-based repacking Taylor Blau
2026-02-25 0:20 ` [RFC PATCH 06/14] repack: track the ODB source via existing_packs Taylor Blau
2026-02-25 0:21 ` Taylor Blau
2026-02-25 0:23 ` Taylor Blau
2026-02-25 0:20 ` [RFC PATCH 01/14] midx: use `string_list` for retained MIDX files Taylor Blau
2026-02-26 20:29 ` Junio C Hamano
2026-02-27 3:02 ` Taylor Blau
2026-02-25 0:21 ` [RFC PATCH 02/14] strvec: introduce `strvec_init_alloc()` Taylor Blau
2026-02-26 20:34 ` Junio C Hamano
2026-02-26 20:58 ` Junio C Hamano
2026-02-27 3:07 ` Taylor Blau
2026-02-25 0:21 ` [RFC PATCH 03/14] midx: use `strvec` for `keep_hashes` Taylor Blau
2026-02-25 0:21 ` Taylor Blau [this message]
2026-02-25 0:21 ` [RFC PATCH 05/14] midx: support custom `--base` for incremental MIDX writes Taylor Blau
2026-02-25 0:21 ` [RFC PATCH 07/14] midx: expose `midx_layer_contains_pack()` Taylor Blau
2026-02-25 0:21 ` [RFC PATCH 08/14] repack-midx: factor out `repack_prepare_midx_command()` Taylor Blau
2026-02-25 0:21 ` [RFC PATCH 09/14] repack-midx: extract `repack_fill_midx_stdin_packs()` Taylor Blau
2026-02-25 0:21 ` [RFC PATCH 10/14] repack-geometry: prepare for incremental MIDX repacking Taylor Blau
2026-02-25 0:21 ` [RFC PATCH 11/14] builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK` Taylor Blau
2026-02-25 0:21 ` [RFC PATCH 12/14] repack: implement incremental MIDX repacking Taylor Blau
2026-02-25 0:22 ` [RFC PATCH 13/14] repack: introduce `--write-midx=incremental` Taylor Blau
2026-02-25 0:22 ` [RFC PATCH 14/14] repack: allow `--write-midx=incremental` without `--geometric` Taylor Blau
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=2d31ea3907ff1965302acb089a594746c8dfb149.1771978829.git.me@ttaylorr.com \
--to=me@ttaylorr$(echo .)com \
--cc=git@vger$(echo .)kernel.org \
--cc=gitster@pobox$(echo .)com \
--cc=newren@gmail$(echo .)com \
--cc=peff@peff$(echo .)net \
--cc=ps@pks$(echo .)im \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox