public inbox for git@vger.kernel.org 
 help / color / mirror / Atom feed

From: "R. Diez" <rdiez-2006@rd10•de>
To: "brian m. carlson" <sandals@crustytoothpaste•net>
Cc: git@vger•kernel.org
Subject: Re: git-fetch takes forever on a slow network link. Can parallel mode help?
Date: Mon, 9 Mar 2026 22:08:31 +0100	[thread overview]
Message-ID: <3ed2d803-5df9-44f4-9427-958d28aa1c46@rd10.de> (raw)
In-Reply-To: <aa39obsSbk9R1mqu@fruit.crustytoothpaste.net>


First of all, thanks for the information about upload-pack etc.

> [...]
> the fact is that the file system is much lower latency and much> faster than the network connection over which data is being sent, so
> that's the case that Git optimizes for.

I wouldn't say that reading sequentially is "optimising". It is just the limitation of a simple implementation. Like I said, with modern SSDs, issuing requests in parallel will be faster even on a local filesystem. That would be a real optimisation then.

Some elderly Unix tools like GNU Make realised long time ago that parallel operation is the way to go. Git itself has realised too, so that it can now work in parallel in certain cases (multiple remote repositories, multiple submodules). So old Unix tools don't count as an excuse!

I think we should clearly point out this deficiency. Git must not be perfect, but I would rather know the limitations upfront. At the very least, that would help me make decisions faster, like investing in some sort of a Git server instead of trying to optimise the SMB/CIFS mount.

And who knows, maybe someone will see this post in the future and decide to implement parallel file operations (async I/O) inside upload-pack and the like.


> rsync would also perform poorly in your case because it's again
> optimized for sending less data over the network than it receives from
> the file system.  Similarly with tar over a network pipe.

rsync would probably look at the file dates and sizes and not transfer everything. There are even some parallel rsync variants designed to overcome high network latencies.

But I don't think rsync is worth the effort for me. I'll just wait a while longer every now and then.


There is one more thing I am curious about. Git does not document how it uses SSH (or at least I couldn't find it in the standard end-user documentation). Git cannot launch a process on the target host over SSH, unless Git is already installed on the remote system. After all, the local system may have a different architecture (like AMD vs ARM), so you cannot copy a binary across. And I haven't seen the requirement that Git must be installed on the remote host when connecting over SSH. In that case, I would have probably seen somewhere a version compatibility table between client and server.

So Git must be accessing files over SSH using the standard SSH file transfer operations. I am guessing that the same latency problem will apply here too, because uploads and downloads over SSH will also be sequential. Is my reasoning correct?

Or does Git attempt to find out whether there is a Git on the other side? What happens if there isn't then?


> [...]
> A `git fetch origin` with that configuration will fetch every branch and
> every tag that points into one of those branches.

OK, thanks. It turns out my repository has no branches at all, so that wouldn't help me anyway.

Best regards,
   rdiez

next prev parent reply	other threads:[~2026-03-09 21:08 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-03-06 20:13 git-fetch takes forever on a slow network link. Can parallel mode help? R. Diez
2026-03-06 20:54 ` brian m. carlson
2026-03-07 21:28   ` R. Diez
2026-03-08  1:44     ` brian m. carlson
2026-03-08 21:08       ` R. Diez
2026-03-08 22:52         ` brian m. carlson
2026-03-09 21:08           ` R. Diez [this message]
2026-03-10 22:50             ` brian m. carlson
2026-03-11 18:05               ` R. Diez

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=3ed2d803-5df9-44f4-9427-958d28aa1c46@rd10.de \
    --to=rdiez-2006@rd10$(echo .)de \
    --cc=git@vger$(echo .)kernel.org \
    --cc=sandals@crustytoothpaste$(echo .)net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox