Subject: Re: [RFC v1] Add virtual file system settings and hook proc

From: Ævar Arnfjörð Bjarmason <[email-masked]>
Date: Mon, 05 Nov 2018 12:40:53 +0100

On Sun, Nov 04 2018, Duy Nguyen wrote:

> On Wed, Oct 31, 2018 at 9:53 PM Ben Peart <[email-masked]> wrote:
>> >> +core.virtualFilesystem::
>> >> +       If set, the value of this variable is used as a command which
>> >> +       will identify all files and directories that are present in
>> >> +       the working directory.  Git will only track and update files
>> >> +       listed in the virtual file system.  Using the virtual file system
>> >> +       will supersede the sparse-checkout settings which will be ignored.
>> >> +       See the "virtual file system" section of linkgit:githooks[6].
>> >
>> > It sounds like "virtual file system" is just one of the use cases for
>> > this feature, which is more about a dynamic source of sparse-checkout
>> > bits. Perhaps name the config key with something along sparse checkout
>> > instead of naming it after a use case.
>>
>> It's more than a dynamic sparse-checkout because the same list is also
>> used to exclude any file/folder not listed.  That means any file not
>> listed won't ever be updated by git (like in 'checkout' for example) so
>> 'stale' files could be left in the working directory.  It also means git
>> won't find new/untracked files unless they are specifically added to the
>> list.
>
> OK. I'm not at all interested in carrying maintenance burden for some
> software behind closed doors. I could see values in having a more
> flexible sparse checkout but this now seems like very tightly designed
> for GVFS. So unless there's another use case (preferably open source)
>  for this, I don't think this should be added in git.git.

I haven't looked at the patch in any detail beyond skimming it, and
you're more familiar with this area...

But in principle I'm very interested in getting something like this in
git.git, even if we were to assume GVFS was a 100% proprietary
implementation (which it's not, it's open source[1] and there's work on
a non-Windows port[2]).

By some definition we already support at least one "virtual filesystem",
or two. I.e. we check files into a *.git repository where everything's
trees & blobs, and then we need to ferry it back & forth between that
and a POSIX-like fs using readdir(), stat(), open() and the like. Of
course this isn't "virtual" in the fs sense, but it's a foreign virtual
FS as far as git is concerned.

Our other "virtual filesystem" (although this is a stretch) is what we
expose via the plumbing commands.

Both of these, as covered at length in various GVFS design docs /
background info, aren't a good fit for some implementation where you'd
like a file-like view that works with the index/status etc. but doesn't
involve checking out everything. There's a huge gap to bridge between
our plumbing & the features we expose that require a POSIX-like fs.

I can see such an ability being very useful for things that aren't the
massive repo Microsoft is interested in supporting. E.g. even for a
~500MB repo it's a stretch to clone that, edit it, and submit a PR on
something like an iOS/Android device.

Once we have something in git to support the likes of GVFS supporting
use-cases like that also becomes easier. Between shallow cloning, the
various object filters & this it would be great to be able to get a 1MB
clone of git.git on my phone and submit a patch to it.

The only potential downside I see is that there's currently exactly one
implementation of this sort of thing in the wild, so we risk any such
API becoming too tied up with just what GVFS wants, and not what we'd
like to support with such a thing in general. This is what e.g. the w3c
tries to avoid with having multiple browser implementations before
something is standardized.

But I think that's easy to work around. E.g. we can document any such
API saying that it's an experimental v1 and is on probation until we get
more users of the feature and figure out the warts & limitations. If it
does end up being too GVFS-specific we could make a breaking change at
some point and eventually drop the v1 version.

1. https://gvfs.io/
2. https://arstechnica.com/gadgets/2017/11/microsoft-and-github-team-up-to-take-git-virtual-file-system-to-macos-linux/