Subject: [PATCH v3 00/12] Introduce `refStorage` extension

From: Patrick Steinhardt <[email-masked]>
Date: Fri, 29 Dec 2023 08:26:22 +0100

Hi,

this is the third version of my patch series that introduces the new
`refStorage` extension. This extension will be used for the upcoming
reftable backend.

Changes compared to v3:

  - The `ref_storage_format` is now tracked as an `unsigned int`,
    proposed by Junio.

  - Reworded the commit message in patch 2, proposed by Eric.

  - Added a NEEDSWORK comment to `get_worktrees_internal()`, propose by
    Eric.

Thanks for your reviews!

Patrick

Patrick Steinhardt (12):
  t: introduce DEFAULT_REPO_FORMAT prereq
  worktree: skip reading HEAD when repairing worktrees
  refs: refactor logic to look up storage backends
  setup: start tracking ref storage format
  setup: set repository's formats on init
  setup: introduce "extensions.refStorage" extension
  setup: introduce GIT_DEFAULT_REF_FORMAT envvar
  t: introduce GIT_TEST_DEFAULT_REF_FORMAT envvar
  builtin/rev-parse: introduce `--show-ref-format` flag
  builtin/init: introduce `--ref-format=` value flag
  builtin/clone: introduce `--ref-format=` value flag
  t9500: write "extensions.refstorage" into config

 Documentation/config/extensions.txt           | 11 +++
 Documentation/git-clone.txt                   |  6 ++
 Documentation/git-init.txt                    |  7 ++
 Documentation/git-rev-parse.txt               |  3 +
 Documentation/git.txt                         |  5 ++
 Documentation/ref-storage-format.txt          |  1 +
 .../technical/repository-version.txt          |  5 ++
 builtin/clone.c                               | 17 ++++-
 builtin/init-db.c                             | 15 +++-
 builtin/rev-parse.c                           |  4 ++
 refs.c                                        | 34 ++++++---
 refs.h                                        |  3 +
 refs/debug.c                                  |  1 -
 refs/files-backend.c                          |  1 -
 refs/packed-backend.c                         |  1 -
 refs/refs-internal.h                          |  1 -
 repository.c                                  |  6 ++
 repository.h                                  |  7 ++
 setup.c                                       | 66 +++++++++++++++--
 setup.h                                       | 10 ++-
 t/README                                      |  3 +
 t/t0001-init.sh                               | 70 +++++++++++++++++++
 t/t1500-rev-parse.sh                          | 17 +++++
 t/t3200-branch.sh                             |  2 +-
 t/t5601-clone.sh                              | 17 +++++
 t/t9500-gitweb-standalone-no-errors.sh        |  5 ++
 t/test-lib-functions.sh                       |  5 ++
 t/test-lib.sh                                 | 15 +++-
 worktree.c                                    | 31 +++++---
 29 files changed, 334 insertions(+), 35 deletions(-)
 create mode 100644 Documentation/ref-storage-format.txt

Range-diff against v2:
 1:  3613439cb7 =  1:  578deaabcf t: introduce DEFAULT_REPO_FORMAT prereq
 2:  ecf4f1ddee !  2:  77c7213c66 worktree: skip reading HEAD when repairing worktrees
    @@ Commit message
         is logic that we resolve their respective worktree HEADs, even though
         that information may not actually be needed in the end by all callers.
     
    -    In the context of git-init(1) this is about to become a problem, because
    -    we do not have a repository that was set up via `setup_git_directory()`
    -    or friends. Consequentially, it is not yet fully initialized at the time
    -    of calling `repair_worktrees()`, and properly setting up all parts of
    -    the repository in `init_db()` before we repair worktrees is not an easy
    -    thing to do. While this is okay right now where we only have a single
    -    reference backend in Git, once we gain a second one we would be trying
    -    to look up the worktree HEADs before we have figured out the reference
    -    format, which does not work.
    +    Although not a problem presently with the file-based reference backend,
    +    it will become a problem with the upcoming reftable backend. In the
    +    context of git-init(1) we do not have a fully-initialized repository set
    +    up via `setup_git_directory()` or friends. Consequently, we do not know
    +    about the repository format when `repair_worktrees()` is called, and
    +    properly setting up all parts of the repositroy in `init_db()` before we
    +    try to repair worktrees is not an easy task. With the introduction of
    +    the reftable backend, we would ultimately try to look up the worktree
    +    HEADs before we have figured out the reference format, which does not
    +    work.
     
         We do not require the worktree HEADs at all to repair worktrees. So
         let's fix this issue by skipping over the step that reads them.
    @@ worktree.c: static void mark_current_worktree(struct worktree **worktrees)
      }
      
     -struct worktree **get_worktrees(void)
    ++/*
    ++ * NEEDSWORK: This function exists so that we can look up metadata of a
    ++ * worktree without trying to access any of its internals like the refdb. It
    ++ * would be preferable to instead have a corruption-tolerant function for
    ++ * retrieving worktree metadata that could be used when the worktree is known
    ++ * to not be in a healthy state, e.g. when creating or repairing it.
    ++ */
     +static struct worktree **get_worktrees_internal(int skip_reading_head)
      {
      	struct worktree **list = NULL;
 3:  12329b99b7 !  3:  47649570bf refs: refactor logic to look up storage backends
    @@ refs.c
     +};
      
     -static struct ref_storage_be *find_ref_storage_backend(const char *name)
    -+static const struct ref_storage_be *find_ref_storage_backend(int ref_storage_format)
    ++static const struct ref_storage_be *find_ref_storage_backend(unsigned int ref_storage_format)
      {
     -	struct ref_storage_be *be;
     -	for (be = refs_backends; be; be = be->next)
    @@ refs.c
      	return NULL;
      }
      
    -+int ref_storage_format_by_name(const char *name)
    ++unsigned int ref_storage_format_by_name(const char *name)
     +{
    -+	for (int i = 0; i < ARRAY_SIZE(refs_backends); i++)
    ++	for (unsigned int i = 0; i < ARRAY_SIZE(refs_backends); i++)
     +		if (refs_backends[i] && !strcmp(refs_backends[i]->name, name))
     +			return i;
     +	return REF_STORAGE_FORMAT_UNKNOWN;
     +}
     +
    -+const char *ref_storage_format_to_name(int ref_storage_format)
    ++const char *ref_storage_format_to_name(unsigned int ref_storage_format)
     +{
     +	const struct ref_storage_be *be = find_ref_storage_backend(ref_storage_format);
     +	if (!be)
    @@ refs.c: static struct ref_store *ref_store_init(struct repository *repo,
      {
     -	const char *be_name = "files";
     -	struct ref_storage_be *be = find_ref_storage_backend(be_name);
    -+	int format = REF_STORAGE_FORMAT_FILES;
    ++	unsigned int format = REF_STORAGE_FORMAT_FILES;
     +	const struct ref_storage_be *be = find_ref_storage_backend(format);
      	struct ref_store *refs;
      
    @@ refs.h: struct string_list;
      struct string_list_item;
      struct worktree;
      
    -+int ref_storage_format_by_name(const char *name);
    -+const char *ref_storage_format_to_name(int ref_storage_format);
    ++unsigned int ref_storage_format_by_name(const char *name);
    ++const char *ref_storage_format_to_name(unsigned int ref_storage_format);
     +
      /*
       * Resolve a reference, recursively following symbolic refererences.
 4:  ddd099fbaf !  4:  837764d0b5 setup: start tracking ref storage format
    @@ refs.c: static struct ref_store *ref_store_init(struct repository *repo,
      					const char *gitdir,
      					unsigned int flags)
      {
    --	int format = REF_STORAGE_FORMAT_FILES;
    +-	unsigned int format = REF_STORAGE_FORMAT_FILES;
     -	const struct ref_storage_be *be = find_ref_storage_backend(format);
     +	const struct ref_storage_be *be;
      	struct ref_store *refs;
    @@ repository.c: void repo_set_hash_algo(struct repository *repo, int hash_algo)
      	repo->hash_algo = &hash_algos[hash_algo];
      }
      
    -+void repo_set_ref_storage_format(struct repository *repo, int format)
    ++void repo_set_ref_storage_format(struct repository *repo, unsigned int format)
     +{
     +	repo->ref_storage_format = format;
     +}
    @@ repository.h: struct repository {
      	const struct git_hash_algo *hash_algo;
      
     +	/* Repository's reference storage format, as serialized on disk. */
    -+	int ref_storage_format;
    ++	unsigned int ref_storage_format;
     +
      	/* A unique-id for tracing purposes. */
      	int trace2_repo_id;
    @@ repository.h: void repo_set_gitdir(struct repository *repo, const char *root,
      		     const struct set_gitdir_args *extra_args);
      void repo_set_worktree(struct repository *repo, const char *path);
      void repo_set_hash_algo(struct repository *repo, int algo);
    -+void repo_set_ref_storage_format(struct repository *repo, int format);
    ++void repo_set_ref_storage_format(struct repository *repo, unsigned int format);
      void initialize_the_repository(void);
      RESULT_MUST_BE_USED
      int repo_init(struct repository *r, const char *gitdir, const char *worktree);
    @@ setup.c: static int is_reinit(void)
      }
      
     -void create_reference_database(const char *initial_branch, int quiet)
    -+void create_reference_database(int ref_storage_format,
    ++void create_reference_database(unsigned int ref_storage_format,
     +			       const char *initial_branch, int quiet)
      {
      	struct strbuf err = STRBUF_INIT;
    @@ setup.c: static void validate_hash_algorithm(struct repository_format *repo_fmt,
      	}
      }
      
    -+static void validate_ref_storage_format(struct repository_format *repo_fmt, int format)
    ++static void validate_ref_storage_format(struct repository_format *repo_fmt,
    ++					unsigned int format)
     +{
     +	if (repo_fmt->version >= 0 &&
     +	    format != REF_STORAGE_FORMAT_UNKNOWN &&
    @@ setup.c: static void validate_hash_algorithm(struct repository_format *repo_fmt,
      int init_db(const char *git_dir, const char *real_git_dir,
     -	    const char *template_dir, int hash, const char *initial_branch,
     +	    const char *template_dir, int hash,
    -+	    int ref_storage_format, const char *initial_branch,
    ++	    unsigned int ref_storage_format,
    ++	    const char *initial_branch,
      	    int init_shared_repository, unsigned int flags)
      {
      	int reinit;
    @@ setup.h: struct repository_format {
      	int worktree_config;
      	int is_bare;
      	int hash_algo;
    -+	int ref_storage_format;
    ++	unsigned int ref_storage_format;
      	int sparse_index;
      	char *work_tree;
      	struct string_list unknown_extensions;
    @@ setup.h: void check_repository_format(struct repository_format *fmt);
      
      int init_db(const char *git_dir, const char *real_git_dir,
      	    const char *template_dir, int hash_algo,
    -+	    int ref_storage_format,
    ++	    unsigned int ref_storage_format,
      	    const char *initial_branch, int init_shared_repository,
      	    unsigned int flags);
      void initialize_repository_version(int hash_algo, int reinit);
     -void create_reference_database(const char *initial_branch, int quiet);
    -+void create_reference_database(int ref_storage_format,
    ++void create_reference_database(unsigned int ref_storage_format,
     +			       const char *initial_branch, int quiet);
      
      /*
 5:  01a1e58a97 =  5:  a51da56d9b setup: set repository's formats on init
 6:  0a586fa648 !  6:  a1e03e4392 setup: introduce "extensions.refStorage" extension
    @@ setup.c: static enum extension_result handle_extension(const char *var,
      		data->hash_algo = format;
      		return EXTENSION_OK;
     +	} else if (!strcmp(ext, "refstorage")) {
    -+		int format;
    ++		unsigned int format;
     +
     +		if (!value)
     +			return config_error_nonbool(var);
    @@ setup.c: static int needs_work_tree_config(const char *git_dir, const char *work
      }
      
     -void initialize_repository_version(int hash_algo, int reinit)
    -+void initialize_repository_version(int hash_algo, int ref_storage_format,
    ++void initialize_repository_version(int hash_algo,
    ++				   unsigned int ref_storage_format,
     +				   int reinit)
      {
      	char repo_version_string[10];
    @@ setup.c: static int create_default_files(const char *template_path,
     
      ## setup.h ##
     @@ setup.h: int init_db(const char *git_dir, const char *real_git_dir,
    - 	    int ref_storage_format,
    + 	    unsigned int ref_storage_format,
      	    const char *initial_branch, int init_shared_repository,
      	    unsigned int flags);
     -void initialize_repository_version(int hash_algo, int reinit);
    -+void initialize_repository_version(int hash_algo, int ref_storage_format,
    ++void initialize_repository_version(int hash_algo,
    ++				   unsigned int ref_storage_format,
     +				   int reinit);
    - void create_reference_database(int ref_storage_format,
    + void create_reference_database(unsigned int ref_storage_format,
      			       const char *initial_branch, int quiet);

 7:  6d8754f73a !  7:  5ffc70e9be setup: introduce GIT_DEFAULT_REF_FORMAT envvar
    @@ Documentation/git.txt: double-quotes and respecting backslash escapes. E.g., the
     
      ## setup.c ##
     @@ setup.c: static void validate_hash_algorithm(struct repository_format *repo_fmt, int hash
    - 
    - static void validate_ref_storage_format(struct repository_format *repo_fmt, int format)
    + static void validate_ref_storage_format(struct repository_format *repo_fmt,
    + 					unsigned int format)
      {
     +	const char *name = getenv("GIT_DEFAULT_REF_FORMAT");
     +
 8:  c645932f3d =  8:  13c074acdf t: introduce GIT_TEST_DEFAULT_REF_FORMAT envvar
 9:  761d647770 =  9:  4ee3c9a2d1 builtin/rev-parse: introduce `--show-ref-format` flag
10:  e382b5bf08 ! 10:  25773e3560 builtin/init: introduce `--ref-format=` value flag
    @@ builtin/init-db.c: int cmd_init_db(int argc, const char **argv, const char *pref
     +	const char *ref_format = NULL;
      	const char *initial_branch = NULL;
      	int hash_algo = GIT_HASH_UNKNOWN;
    -+	int ref_storage_format = REF_STORAGE_FORMAT_UNKNOWN;
    ++	unsigned int ref_storage_format = REF_STORAGE_FORMAT_UNKNOWN;
      	int init_shared_repository = -1;
      	const struct option init_db_options[] = {
      		OPT_STRING(0, "template", &template_dir, N_("template-directory"),
11:  257233658d ! 11:  3f1cb6b9e5 builtin/clone: introduce `--ref-format=` value flag
    @@ builtin/clone.c: int cmd_clone(int argc, const char **argv, const char *prefix)
      	int submodule_progress;
      	int filter_submodules = 0;
      	int hash_algo;
    -+	int ref_storage_format = REF_STORAGE_FORMAT_UNKNOWN;
    ++	unsigned int ref_storage_format = REF_STORAGE_FORMAT_UNKNOWN;
      	const int do_not_override_repo_unix_permissions = -1;
      
      	struct transport_ls_refs_options transport_ls_refs_options =
12:  b8cd06ec53 = 12:  2e7682b2f3 t9500: write "extensions.refstorage" into config

base-commit: e79552d19784ee7f4bbce278fe25f93fbda196fa
-- 
2.43.GIT