Subject: Future structure of submodules and .git/, .git/modules/* organization

From: Ævar Arnfjörð Bjarmason <[email-masked]>
Date: Wed, 14 Apr 2021 12:32:17 +0200

On Fri, Apr 09 2021, Emily Shaffer wrote:

> I'm hoping to work on some other submodule-centric stuff over the coming
> months, and it might end up being very useful to be able to tell "am I a
> submodule?" and "how do I talk to my superproject?" more generally - so
> I'm really open to figuring out a better way than this, if folks have
> ideas.
>
> Patch 1 is a small refactor that we can take or leave - I found
> "SCOPE_SUBMODULE" to be pretty ambiguous, especially since it seems to
> refer to configs from .gitmodules. Even though I decided that
> "superproject" was a better name than "submodule" I still wasn't super
> happy with the ambiguity. But we can drop it if folks don't want to
> rename.

This is less on your patch, and more on the larger work you're
suggesting, but the two are kind of related. Skip to the paragraph
starting with "But why" below for the relevance :)

I very much wish that we could eventually make the use of submodules
totally transparent, i.e. (taking the example of git.git):

 * You clone, and we just get objects from
   https://github.com/cr-marcstevens/sha1collisiondetection.git too

 * The fact that we have:

   160000 commit 855827c583bc30645ba427885caa40c5b81764d2  sha1collisiondetection

   Would become totally invisible to most users unless they run some
   gutsy ls-tree/files comand.

   We used to have a full git dir at sha1collisiondetection/.git and all
   the UX issues that entailed (e.g. switching to an old commit without
   the submodule).

   Now it's a stub and the actual repo is at
   .git/modules/sha1collisiondetection/, so we're kind of partially
   there.

 * I would think that the next (but big) logical step would be to use
   some combination of delta islands, upcoming sparse indexes etc. to
   actually share the object stores of the parent and submodule.

   Things like "git fsck" which now just punt on COMMIT would need to
   become smarter, but e.g. we could repack (or not, with islands)
   between parent and submodule.

I would think that this end goal makes more sense than the current
status quo of teaching every command that needs to e.g. grep the tree to
have a "--recurse-submodules". The distinction would be invisible to the
likes of "git-grep".

It would mean more complexity in e.g. "git commit", but we can imagine
if you wanted a cross-submodule commit it could do those commits
recursively, update parent COMMIT entries etc. (and even, optionally,
push out the submodule changes). That particular thing being so ad-hoc
is a *very* frequent pain point in submodule use.

But why am I talking about this here when all you're suggesting is
another config level?

Well, I think (but have not carefully thought about) that this
CONFIG_SCOPE_GITMODULES is probably a narrow net improvement now. If you
set most options in your .git/config to you that's the same logical
project, why shouldn't you get your diff setting or whatever because you
cd'd to a submodule "in the same project" (from the view of the user).

But I think that for a wider "improve submodules" effort it's worth
someone (and right now, that sounds like it's you) thinking about where
we're going with the feature. Maybe with some technical doc identifying
the most common pain points, what we propose (or could envision) doing
about them.

So e.g. in this case, having per-submodule config could be a step
forward, but it could also be one more step of walking in a circle.

I.e. don't think any user asked for or wanted to stitch together
multiple .git directories into one linked pseudo-checkout, that's
ultimately something we're exposing as an implementation detail. If we
no longer expose that implementation detail, would we be stuck
supporting what's ultimately a workaround feature?

None of that means we shouldn't have that one step forward that solves
real problems today.

But I think we should think about the end goal(s) sooner than
later. E.g. in your case, do you *really* want another config level, or
is it just the easiest way to get what you actually want, which is for a
"git config" in the submodule dir to perhaps consider its .git/config
and .git/modules/sha1collisiondetection/config as the same file for the
purposes of config parsing? Sans things like the remote URLs etc.