Subject: Re: [PATCH v3 2/4] pack-refs: teach --exclude option to exclude refs from being packed

From: Taylor Blau <[email-masked]>
Date: Thu, 11 May 2023 20:00:24 -0400

On Thu, May 11, 2023 at 06:10:32PM +0000, John Cai via GitGitGadget wrote:
> From: John Cai <[email-masked]>
>
> At GitLab, we have a system that creates ephemeral internal refs that
> don't live long before getting deleted. Having an option to exclude
> certain refs from a packed-refs file allows these internal references to
> be deleted much more efficiently.
>
> Add an --exclude option to the pack-refs builtin, and use the ref
> exclusions API to exclude certain refs from being packed into the final
> packed-refs file
>
> Signed-off-by: John Cai <[email-masked]>
> ---
>  Documentation/git-pack-refs.txt | 12 +++++++++++-
>  builtin/pack-refs.c             | 20 ++++++++++++++++----
>  refs.c                          |  4 ++--
>  refs.h                          |  7 ++++++-
>  refs/debug.c                    |  4 ++--
>  refs/files-backend.c            | 16 ++++++++++------
>  refs/packed-backend.c           |  2 +-
>  refs/refs-internal.h            |  3 ++-
>  revision.h                      |  2 +-
>  t/helper/test-ref-store.c       |  3 ++-
>  t/t3210-pack-refs.sh            | 16 ++++++++++++++++
>  11 files changed, 69 insertions(+), 20 deletions(-)
>
> diff --git a/Documentation/git-pack-refs.txt b/Documentation/git-pack-refs.txt
> index e011e5fead3..c0f7426e519 100644
> --- a/Documentation/git-pack-refs.txt
> +++ b/Documentation/git-pack-refs.txt
> @@ -8,7 +8,7 @@ git-pack-refs - Pack heads and tags for efficient repository access
>  SYNOPSIS
>  --------
>  [verse]
> -'git pack-refs' [--all] [--no-prune]
> +'git pack-refs' [--all] [--no-prune] [--exclude <pattern>]
>
>  DESCRIPTION
>  -----------
> @@ -60,6 +60,16 @@ interests.
>  The command usually removes loose refs under `$GIT_DIR/refs`
>  hierarchy after packing them.  This option tells it not to.
>
> +--exclude <pattern>::
> +
> +Do not pack refs matching the given `glob(7)` pattern. Repetitions of this option
> +accumulate exclusion patterns. Use `--no-exclude` to clear and reset the list of
> +patterns. If a ref is already packed, including it with `--exclude` will not
> +unpack it.
> +
> +When used with `--all`, it will use the difference between the set of all refs,
> +and what is provided to `--exclude`.
> +

I think this last paragraph could be simplified, though feel free to
discard my suggestion if you think it makes things less clear.

  When used with `--all`, pack only loose refs which do not match any of
  the provided `--exclude` patterns.

>  int cmd_pack_refs(int argc, const char **argv, const char *prefix)
>  {
>  	unsigned int flags = PACK_REFS_PRUNE;
> +	static struct ref_exclusions excludes = REF_EXCLUSIONS_INIT;
> +	struct pack_refs_opts pack_refs_opts = {.exclusions = &excludes, .flags = flags};
> +	static struct string_list option_excluded_refs = STRING_LIST_INIT_NODUP;
> +	struct string_list_item *item;

Since this list does not appear to be sensitive to its order, have you
considered using the strvec API instead of the string_list one?

> diff --git a/refs.c b/refs.c
> index d2a98e1c21f..881a0da65cf 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -2132,9 +2132,9 @@ void base_ref_store_init(struct ref_store *refs, struct repository *repo,
>  }
>
>  /* backend functions */
> -int refs_pack_refs(struct ref_store *refs, unsigned int flags)
> +int refs_pack_refs(struct ref_store *refs, struct pack_refs_opts *opts)
>  {
> -	return refs->be->pack_refs(refs, flags);
> +	return refs->be->pack_refs(refs, opts);
>  }
>
>  int peel_iterated_oid(const struct object_id *base, struct object_id *peeled)
> diff --git a/refs.h b/refs.h
> index 123cfa44244..46020bd335c 100644
> --- a/refs.h
> +++ b/refs.h
> @@ -63,6 +63,11 @@ struct worktree;
>  #define RESOLVE_REF_NO_RECURSE 0x02
>  #define RESOLVE_REF_ALLOW_BAD_NAME 0x04
>
> +struct pack_refs_opts {
> +	unsigned int flags;
> +	struct ref_exclusions *exclusions;

I think this would be OK to include directly in the struct instead of
via a pointer, but either is fine.

> @@ -1175,15 +1176,18 @@ static void prune_refs(struct files_ref_store *refs, struct ref_to_prune **refs_
>   */
>  static int should_pack_ref(const char *refname,
>  			   const struct object_id *oid, unsigned int ref_flags,
> -			   unsigned int pack_flags)
> +			   struct pack_refs_opts *opts)
>  {
>  	/* Do not pack per-worktree refs: */
>  	if (parse_worktree_ref(refname, NULL, NULL, NULL) !=
>  	    REF_WORKTREE_SHARED)
>  		return 0;
>
> +	if (opts->exclusions && ref_excluded(opts->exclusions, refname))
> +		return 0;

Looks good, here is where we throw out refs that we don't want. I wonder
if ref_excluded() does the right thing with a zero-initialized argument
(i.e. that it behaves as if nothing matches).

I wonder if it's possible to skip over certain loose references by
avoiding traversal into the sub-directories for simple prefixes. That
may be a premature optimization, though, so I don't think you
necessarily need to worry about it in this round.

> +test_expect_success 'test excluded refs are not packed' '
> +	git branch dont_pack1 &&
> +	git branch dont_pack2 &&
> +	git branch pack_this &&
> +	git pack-refs --all --exclude "refs/heads/dont_pack*" &&
> +	test -f .git/refs/heads/dont_pack1 &&
> +	test -f .git/refs/heads/dont_pack2 &&
> +	! test -f ./git/refs/heads/pack_this'
> +
> +test_expect_success 'test --no-exclude refs clears excluded refs' '
> +	git branch dont_pack3 &&
> +	git branch dont_pack4 &&
> +	git pack-refs --all --exclude "refs/heads/dont_pack*" --no-exclude &&
> +	! test -f .git/refs/heads/dont_pack3 &&
> +	! test -f .git/refs/heads/dont_pack4'

Tests look good. The trailing quote is a little odd to be placed on the
last line of the test instead of off on its own, but I suppose that is
imitating existing style, which is OK.

Thanks,
Taylor