Subject: Re: [RFC] On the --depth argument when fetching with submodules

From: Jonathan Nieder <[email-masked]>
Date: Mon, 8 Feb 2016 12:38:20 -0800

Hi,

Stefan Beller wrote:

> 1) You work on the superproject and don't care about the submodules.
> In this case you want the superproject non-shallow and the submodules
> may be just fine with depth 1. (Think of libraries pulled in via Git instead
> of via the build system)
>
> 2) The superproject is a collection of submodules, i.e. not much content
> in the superproject except for the submodules. You want to work
> in the submodules, i.e. you want the suberproject shallow, and all
> submodules deep.
>
> 3) same as 2, but you're interested in only one (or a few) submodules,
> which means you want superproject and most of the other submodules
> shallow, but one submodule needs to be deep.
>
> So covering 1 and 2 is easy, 3 is complicated.
> For 1) we can make it so, that the depth argument is not passed on,
> but only covers the referenced submodule commits, and then we
> introduce another switch "--submodule-depth" to cover 2).
>
> For 3 we don't know which submodules the user is interested in,
> so the user needs to unshallow the interesting submodules themselves
> after doing a "--depth 1 --submodule--depth 1" clone. "--depth 1" sort of
> implies "--submodule--depth 1", though.

What does a --submodule-depth other than 1 mean?  For example, if I pass 2
there, does that mean I want depth 2 in all sub-sub-submodules, and
sub-sub-sub-submodules, etc?  This has similar problems to the existing
"--depth=2 --recurse-submodules" options.

If I understood the use cases you are describing correctly, you'd want two
relevant options when cloning:

 --depth: how deep of a history to fetch in the superproject
 --shallow-submodules: whether the history in submodules is interesting

If the history in a submodule is uninteresting and the submodule
supports fetch-by-sha1, then a fetch-by-sha1 with depth 1 for any
commits needed by the superproject would get all objects needed.

--depth could imply --shallow-submodules in the same way as it implies
--single-branch because when people pass --depth they are usually
trying to avoid transferring too much data (for disk space or clone
time reasons).  Then someone wanting non-shallow submodules can
explicitly pass --no-shallow-submodules with --depth.

[...]
> So for fetching I think we need to have a "--try-to-get-commit <sha1>"
> argument for fetch, which depending on the server capabilities and
> the history obtained otherwise may try again to fetch the exact sha1.

The case of servers without fetch-by-sha1 is a hard one.

 1. One approach would be to use the ls-remote response to see if
    there is a ref pointing to the particular SHA-1 needed.  If there
    is one, we're all set.

 2. If there is no ref pointing to the desired SHA-1, fetching HEAD
    (or the branch named in the submodule.<name>.branch configuration)
    with full history is possible to work as a fallback.

 3. If the history of HEAD also doesn't contain the desired SHA-1,
    fetching the refs named in the default refspec (refs/heads/*,
    refs/tags/*) might work.

 4. If even then the desired SHA-1 is unreachable, then there's
    nothing left to try --- it's time to error out and have the user
    contact their server admin (as they'd already be doing today
    without --depth).

Perhaps some people would want to error out earlier.  Some
configuration or a --strict-shallow-submodules option could be useful
to error out if step (1) fails.  That would allow the user to be sure
that the submodules are actually fetched without history.

The missing functionality from fetch/ls-remote/etc is a way to read
the server capabilities that were sent with the ref listing to tell
whether the server supports fetch-by-sha1.

> On Sun, Feb 7, 2016 at 5:32 AM, Lars Schneider <[email-masked]> wrote:

>> On top of that I think Git's error message is really confusing if
>> you clone a repo with "--depth" that has submodules and Git is not
>> fetching the necessary submodule commits:
>>
>> Unable to checkout '$SHA' in submodule path 'path/to/submodule'

Yep.  Thanks for working on it.

This confusing error can appear without --depth, too (if the commit
named in the submodule is not an ancestor of anything in refs/heads/*
or refs/tags/*).  For example, this happens in gerrit when people try
to check out a change under review in a parent project that imports a
change in a subproject that has not been accepted yet (it is reachable
from refs/changes/* but not refs/heads/*).

Thanks,
Jonathan