Subject: [PATCH v2 0/4] Fix a few split-index bugs

From: "Johannes Schindelin via GitGitGadget" <[email-masked]>
Date: Sun, 26 Mar 2023 22:45:39 +0000

I received an internal bug report that after upgrading from v2.39.2 to
v2.40.0, some users ran into the following error message:

BUG: fsmonitor.c:21: fsmonitor_dirty has more entries than the index (57 > 0)

It sounds very much like the report we received in
https://lore.kernel.org/git/[email-masked]/,
but sadly that thread petered out when the reporter stopped being able to
reproduce the problem.

After a few days of investigating, I am convinced that this is due to some
old bugs, and not actually a regression in v2.40.0 (although I can believe
that some improvements in v2.40.0 might make it easier to run into these
bugs).

This patch series addresses those bugs.

Note: While the Git maintainer has stated a strong preference to introduce
regression tests in the same patch that fixes the corresponding regression,
this patch series starts with a stand-alone patch that demonstrates a
problematic scenario via a new test_expect_failure test case. The reason why
I specifically split out the test into its own commit is that there is a lot
of information to unpack in the commit message that is larger than any of
the subsequent bug fixes. Besides, it motivates not only the second patch
(which marks the test case as test_expect_success) but paints the larger
picture necessary to understand also the need for the remaining two patches.

This patch series is based on maint-2.37, the oldest maintenance branch it
applies without merge conflicts. When merging with next, there are only
trivial conflicts in unpack-trees.c due to en/dir-api-cleanup where
o->result is now o->internal.result.

Changes since v1:

 * Fix a double "the" in a commit message
 * Replace enum strip_extensions by the bit field enum write_extensions,
   inverting the meaning of the values to avoid double negatives
 * Leave a trailing comma at the definition of the enum values

Johannes Schindelin (4):
  split-index & fsmonitor: demonstrate a bug
  split-index; stop abusing the `base_oid` to strip the "link" extension
  fsmonitor: avoid overriding `cache_changed` bits
  unpack-trees: take care to propagate the split-index flag

 fsmonitor.h                  |  2 +-
 read-cache.c                 | 49 +++++++++++++++++++++++-------------
 t/t7527-builtin-fsmonitor.sh | 37 +++++++++++++++++++++++++++
 unpack-trees.c               |  2 ++
 4 files changed, 72 insertions(+), 18 deletions(-)

base-commit: eb88fe1ff5ceb34845f0919b8bdc60d8a1703cf6
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1497%2Fdscho%2Ffix-split-index-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1497/dscho/fix-split-index-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/1497

Range-diff vs v1:

 1:  c025fccbdde = 1:  c025fccbdde split-index & fsmonitor: demonstrate a bug
 2:  f1897b88072 ! 2:  8cc075f6325 split-index; stop abusing the `base_oid` to strip the "link" extension
     @@ Commit message
      
          One might be tempted to simply call `discard_split_index()` instead,
          under the assumption that Git decided to write a non-split index and
     -    therefore the the `split_index` structure might no longer be wanted.
     +    therefore the `split_index` structure might no longer be wanted.
          However, that is not possible because that would release index entries
          in `split_index->base` that are likely to still be in use. Therefore we
          cannot do that.
      
     -    The next best thing we _can_ do is to introduce a flag, specifically
     -    indicating when the "link" extension should be skipped. So that's what
     -    we do here.
     +    The next best thing we _can_ do is to introduce a bit field to indicate
     +    specifically which index extensions (not) to write. So that's what we do
     +    here.
      
          Signed-off-by: Johannes Schindelin <[email-masked]>
      
     @@ read-cache.c: static int record_ieot(void)
       	return !git_config_get_index_threads(&val) && val != 1;
       }
       
     -+enum strip_extensions {
     -+	WRITE_ALL_EXTENSIONS = 0,
     -+	STRIP_ALL_EXTENSIONS = 1,
     -+	STRIP_LINK_EXTENSION_ONLY = 2
     ++enum write_extensions {
     ++	WRITE_NO_EXTENSION =              0,
     ++	WRITE_SPLIT_INDEX_EXTENSION =     1<<0,
     ++	WRITE_CACHE_TREE_EXTENSION =      1<<1,
     ++	WRITE_RESOLVE_UNDO_EXTENSION =    1<<2,
     ++	WRITE_UNTRACKED_CACHE_EXTENSION = 1<<3,
     ++	WRITE_FSMONITOR_EXTENSION =       1<<4,
      +};
     ++#define WRITE_ALL_EXTENSIONS ((enum write_extensions)-1)
      +
       /*
        * On success, `tempfile` is closed. If it is the temporary file
     @@ read-cache.c: static int record_ieot(void)
        */
       static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
      -			  int strip_extensions, unsigned flags)
     -+			  enum strip_extensions strip_extensions, unsigned flags)
     ++			  enum write_extensions write_extensions, unsigned flags)
       {
       	uint64_t start = getnanotime();
       	struct hashfile *f;
     @@ read-cache.c: static int do_write_index(struct index_state *istate, struct tempf
       	}
       
      -	if (!strip_extensions && istate->split_index &&
     -+	if (strip_extensions == WRITE_ALL_EXTENSIONS && istate->split_index &&
     - 	    !is_null_oid(&istate->split_index->base_oid)) {
     +-	    !is_null_oid(&istate->split_index->base_oid)) {
     ++	if (write_extensions & WRITE_SPLIT_INDEX_EXTENSION &&
     ++	    istate->split_index) {
       		struct strbuf sb = STRBUF_INIT;
       
     + 		if (istate->sparse_index)
      @@ read-cache.c: static int do_write_index(struct index_state *istate, struct tempfile *tempfile,
       		if (err)
       			return -1;
       	}
      -	if (!strip_extensions && !drop_cache_tree && istate->cache_tree) {
     -+	if (strip_extensions != STRIP_ALL_EXTENSIONS && !drop_cache_tree && istate->cache_tree) {
     ++	if (write_extensions & WRITE_CACHE_TREE_EXTENSION &&
     ++	    !drop_cache_tree && istate->cache_tree) {
       		struct strbuf sb = STRBUF_INIT;
       
       		cache_tree_write(&sb, istate->cache_tree);
     @@ read-cache.c: static int do_write_index(struct index_state *istate, struct tempf
       			return -1;
       	}
      -	if (!strip_extensions && istate->resolve_undo) {
     -+	if (strip_extensions != STRIP_ALL_EXTENSIONS && istate->resolve_undo) {
     ++	if (write_extensions & WRITE_RESOLVE_UNDO_EXTENSION &&
     ++	    istate->resolve_undo) {
       		struct strbuf sb = STRBUF_INIT;
       
       		resolve_undo_write(&sb, istate->resolve_undo);
     @@ read-cache.c: static int do_write_index(struct index_state *istate, struct tempf
       			return -1;
       	}
      -	if (!strip_extensions && istate->untracked) {
     -+	if (strip_extensions != STRIP_ALL_EXTENSIONS && istate->untracked) {
     ++	if (write_extensions & WRITE_UNTRACKED_CACHE_EXTENSION &&
     ++	    istate->untracked) {
       		struct strbuf sb = STRBUF_INIT;
       
       		write_untracked_extension(&sb, istate->untracked);
     @@ read-cache.c: static int do_write_index(struct index_state *istate, struct tempf
       			return -1;
       	}
      -	if (!strip_extensions && istate->fsmonitor_last_update) {
     -+	if (strip_extensions != STRIP_ALL_EXTENSIONS && istate->fsmonitor_last_update) {
     ++	if (write_extensions & WRITE_FSMONITOR_EXTENSION &&
     ++	    istate->fsmonitor_last_update) {
       		struct strbuf sb = STRBUF_INIT;
       
       		write_fsmonitor_extension(&sb, istate);
     @@ read-cache.c: static int commit_locked_index(struct lock_file *lk)
       		return commit_lock_file(lk);
       }
       
     -+/*
     -+ * Write the Git index into a `.lock` file
     -+ *
     -+ * If `strip_link_extension` is non-zero, avoid writing any "link" extension
     -+ * (used by the split-index feature).
     -+ */
     - static int do_write_locked_index(struct index_state *istate, struct lock_file *lock,
     +-static int do_write_locked_index(struct index_state *istate, struct lock_file *lock,
      -				 unsigned flags)
     -+				 unsigned flags, int strip_link_extension)
     ++static int do_write_locked_index(struct index_state *istate,
     ++				 struct lock_file *lock,
     ++				 unsigned flags,
     ++				 enum write_extensions write_extensions)
       {
       	int ret;
       	int was_full = istate->sparse_index == INDEX_EXPANDED;
     @@ read-cache.c: static int do_write_locked_index(struct index_state *istate, struc
       	trace2_region_enter_printf("index", "do_write_index", the_repository,
       				   "%s", get_lock_file_path(lock));
      -	ret = do_write_index(istate, lock->tempfile, 0, flags);
     -+	ret = do_write_index(istate, lock->tempfile, strip_link_extension ? STRIP_LINK_EXTENSION_ONLY : 0, flags);
     ++	ret = do_write_index(istate, lock->tempfile, write_extensions, flags);
       	trace2_region_leave_printf("index", "do_write_index", the_repository,
       				   "%s", get_lock_file_path(lock));
       
     @@ read-cache.c: static int write_split_index(struct index_state *istate,
       	int ret;
       	prepare_to_write_split_index(istate);
      -	ret = do_write_locked_index(istate, lock, flags);
     -+	ret = do_write_locked_index(istate, lock, flags, 0);
     ++	ret = do_write_locked_index(istate, lock, flags, WRITE_ALL_EXTENSIONS);
       	finish_writing_split_index(istate);
       	return ret;
       }
     +@@ read-cache.c: static int write_shared_index(struct index_state *istate,
     + 
     + 	trace2_region_enter_printf("index", "shared/do_write_index",
     + 				   the_repository, "%s", get_tempfile_path(*temp));
     +-	ret = do_write_index(si->base, *temp, 1, flags);
     ++	ret = do_write_index(si->base, *temp, WRITE_NO_EXTENSION, flags);
     + 	trace2_region_leave_printf("index", "shared/do_write_index",
     + 				   the_repository, "%s", get_tempfile_path(*temp));
     + 
      @@ read-cache.c: int write_locked_index(struct index_state *istate, struct lock_file *lock,
       	if ((!si && !test_split_index_env) ||
       	    alternate_index_output ||
     @@ read-cache.c: int write_locked_index(struct index_state *istate, struct lock_fil
      -		if (si)
      -			oidclr(&si->base_oid);
      -		ret = do_write_locked_index(istate, lock, flags);
     -+		ret = do_write_locked_index(istate, lock, flags, 1);
     ++		ret = do_write_locked_index(istate, lock, flags,
     ++					    ~WRITE_SPLIT_INDEX_EXTENSION);
       		goto out;
       	}
       
     @@ read-cache.c: int write_locked_index(struct index_state *istate, struct lock_fil
       		if (!temp) {
      -			oidclr(&si->base_oid);
      -			ret = do_write_locked_index(istate, lock, flags);
     -+			ret = do_write_locked_index(istate, lock, flags, 1);
     ++			ret = do_write_locked_index(istate, lock, flags,
     ++						    ~WRITE_SPLIT_INDEX_EXTENSION);
       			goto out;
       		}
       		ret = write_shared_index(istate, &temp, flags);
 3:  c1c35f0f026 = 3:  89b3cd9a668 fsmonitor: avoid overriding `cache_changed` bits
 4:  3963d3e5428 = 4:  df61146eaf5 unpack-trees: take care to propagate the split-index flag

-- 
gitgitgadget