From: <rsbecker@nexbridge•com>
To: "'ellie'" <el@horse64•org>, <git@vger•kernel.org>
Subject: RE: With big repos and slower connections, git clone can be hard to work with
Date: Mon, 8 Jul 2024 13:38:34 -0400 [thread overview]
Message-ID: <003d01dad15d$abd3f750$037be5f0$@nexbridge.com> (raw)
In-Reply-To: <ff50859c-8e67-46ed-a8bd-ad7f836374c1@horse64.org>
On Monday, July 8, 2024 1:06 PM, ellie wrote:
>On 7/8/24 6:23 PM, rsbecker@nexbridge•com wrote:
>> On Monday, July 8, 2024 11:49 AM, ellie wrote:
>>> On 7/8/24 5:31 PM, rsbecker@nexbridge•com wrote:
>>>> On Monday, July 8, 2024 11:15 AM, ellie wrote:
>>>>> On 7/8/24 4:32 PM, Konstantin Khomoutov wrote:
>>>>>> On Mon, Jul 08, 2024 at 04:28:25AM +0200, ellie wrote:
>>>>>>
>>>>>> [...]
>>>>>>> error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly:
>>>>>>> CANCEL (err 8)
>>>>>> [...]
>>>>>>> It seems extremely unlikely to me to be possibly an ISP issue,
>>>>>>> for which I already listed the reasons. An additional one is
>>>>>>> HTTPS downloads from github outside of git, e.g. from zip
>>>>>>> archives, for way larger files work fine as well.
>>>>>> [...]
>>>>>>
>>>>>> What if you explicitly disable HTTP/2 when cloning?
>>>>>>
>>>>>> git -c http.version=HTTP/1.1 clone ...
>>>>>>
>>>>>> should probably do this.
>>>>>>
>>>>>
>>>>> Thanks for the idea! I tested it:
>>>>>
>>>>> $ git -c http.version=HTTP/1.1 clone
>>>>> https://github.com/maliit/keyboard
>>>>> maliit-keyboard
>>>>> Cloning into 'maliit-keyboard'...
>>>>> remote: Enumerating objects: 23243, done.
>>>>> remote: Counting objects: 100% (464/464), done.
>>>>> remote: Compressing objects: 100% (207/207), done.
>>>>> error: RPC failed; curl 18 transfer closed with outstanding read
>>>>> data remaining
>>>>> error: 5361 bytes of body are still expected
>>>>> fetch-pack: unexpected disconnect while reading sideband packet
>>>>> fatal: early EOF
>>>>> fatal: fetch-pack: invalid index-pack output
>>>>>
>>>>> Sadly, it seems like the error is only slightly different. It was
>>>>> still worth a try. I contacted GitHub support a while ago but it
>>>>> got stuck. If there were resume available such hiccups wouldn't
>>>>> matter, I hope that explains why I suggested that feature.
>>>>
>>>> I don't really understand what "it got stuck" means. Is that a
>>>> colloquialism? What
>>> got stuck? That case at GitHub?
>>>>
>>>> Have you tried git config --global http.postBuffer 524288000
>>>>
>>>> It might help. The feature being requesting, even if possible, will
>>>> probably not
>>> happen quickly, unless someone has a solid and simple design for
>>> this. That is why we are trying to figure out the root cause of your
>>> situation, which is not clear to me as to what exactly is failing (possibly a buffer
>size issue, if this is consistently failing).
>>> My experience, as I said before, on these symptoms, is a proxy (even
>>> a local one) that is in the way. If you have your linux instance on a
>>> VM, the hypervisor may not be configured correctly. Lack of further
>>> evidence (all we really have is the curl RPC
>>> failure) makes diagnosing this very difficult.
>>>>
>>>
>>> Thanks for your response, I appreciate it. I don't know what the hold
>>> up is for them, but I'm probably too unimportant, which I understand.
>>> I'm not an enterprise user, and >99% of others have faster
>>> connections than me which is perhaps why they dodge this config(?) issue.
>>>
>>> And thanks for your suggestion, but sadly it seems to have no effect:
>>>
>>> $ git config --global http.postBuffer 524288000 $ git -c
>>> http.version=HTTP/1.1 clone https://github.com/maliit/keyboard
>>> maliit-keyboard
>>> Cloning into 'maliit-keyboard'...
>>> remote: Enumerating objects: 23243, done.
>>> remote: Counting objects: 100% (464/464), done.
>>> remote: Compressing objects: 100% (207/207), done.
>>> error: RPC failed; curl 18 transfer closed with outstanding read data
>>> remaining
>>> error: 2444 bytes of body are still expected
>>> fetch-pack: unexpected disconnect while reading sideband packet
>>> fatal: early EOF
>>> fatal: fetch-pack: invalid index-pack output
>>>
>>> I'm doubtful this is solvable without either some resume or a fix from Github's
>end.
>>> But I can use SSH clone so this isn't urgent.
>>>
>>> Resume just seemed like an idea that would also help others, and it's
>>> what makes many other internet services work much better for me.
>>
>> I do not know which pack file is having the issue - it may be the first one. Try
>running with the following environment variables GIT_TRACE=true and
>GIT_PACKET_TRACE=true. This will not correct the problem but might give
>additional helpful information. git uses libcurl to perform https transfers - which
>appears to be where the error is coming from. It is my opinion, given the issue is
>very likely in curl, that a restart capability will not help at all - at least not until we
>find the actual root cause (still mostly an unknown, although this error is widely
>discussed online in other non-git places). The failure appears to be transferring a
>single pack file (139824442 bytes) size may be an issue, but restarting in the middle
>of a pack file may not solve the problem (discussed in other threads) as the file is
>potentially built on demand (as I understand it from GitHub) and may not be the
>same on the next clone attempt. What we probably will find is that a restart will be
>stuck in the same spot and not move forward because the failure is not at a file
>boundary.
>>
>> In addition to this, GitHub may have limits on the size of files that can be
>transferred, which you might be hitting (unlikely but possible). Check your plan
>options. I tried on a light plan, so this is unlikely but I want to exclude it.
>>
>>
>I attached the output of this command:
>
>$ GIT_TRACE=true GIT_PACKET_TRACE=true git -c http.version=HTTP/1.1 clone
>https://github.com/malii t/keyboard maliit-keyboard > log.txt 2>&1
>
>My best guess is still that due to some unfortunate timeout choice, Github's end
>simply becomes impatient and closes the connection.
18:44:33.182907 git.c:465 trace: built-in: git clone https://github.com/maliit/keyboard maliit-keyboard
Cloning into 'maliit-keyboard'...
18:44:33.186926 run-command.c:657 trace: run_command: git remote-https origin https://github.com/maliit/keyboard
18:44:33.188668 git.c:750 trace: exec: git-remote-https origin https://github.com/maliit/keyboard
18:44:33.188728 run-command.c:657 trace: run_command: git-remote-https origin https://github.com/maliit/keyboard
18:44:34.757740 run-command.c:657 trace: run_command: git index-pack --stdin --fix-thin '--keep=fetch-pack 14261 on elliedeck' --check-self-contained-and-connected
18:44:34.759305 git.c:465 trace: built-in: git index-pack --stdin --fix-thin '--keep=fetch-pack 14261 on elliedeck' --check-self-contained-and-connected
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
error: 5858 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
From what I could tell from the log, the operation took less than 3 seconds. How long does it appear to take to you? This does not look like a timeout. In fact, it looks like the failure happened before git was able to process any content. From what I read from the log, libcurl encountered a failure and passed that up to git, which stopped the operation. You could try putting -v into your .curlrc file or otherwise getting some verbose information out of curl here the failure is occurring. I would also suggest passing this over to the curl team for examination. I am at a loss on resolving this, further, particularly if there are no intermediary components like firewalls and proxies - note that many ISPs build firewalls and proxies into their NAT routers. A curl verbose trace might show this. My home ISP in Canada has all kinds of stuff in their cable modems, which I had disabled by the tech who installed the box, and I have no issues cloning the above repo. They do have QoS limits but have not blocked https downloads.
--Randall
next prev parent reply other threads:[~2024-07-08 17:38 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-06-07 23:28 With big repos and slower connections, git clone can be hard to work with ellie
2024-06-07 23:33 ` rsbecker
2024-06-08 0:03 ` ellie
2024-06-08 0:35 ` rsbecker
2024-06-08 0:46 ` ellie
2024-06-08 8:43 ` Jeff King
2024-06-08 9:40 ` ellie
2024-06-08 9:44 ` ellie
2024-06-08 10:38 ` Jeff King
2024-06-08 10:35 ` Jeff King
2024-06-08 11:05 ` ellie
2024-06-08 19:00 ` Junio C Hamano
2024-06-08 20:16 ` ellie
2024-06-10 6:46 ` Patrick Steinhardt
2024-06-10 19:04 ` Emily Shaffer
2024-06-10 20:34 ` Junio C Hamano
2024-06-10 21:55 ` ellie
2024-06-13 10:10 ` Toon claes
2024-06-11 6:31 ` Jeff King
2024-06-11 15:12 ` Junio C Hamano
2024-06-29 1:53 ` Sitaram Chamarty
2024-06-11 6:26 ` Jeff King
2024-06-11 19:40 ` Ivan Frade
2024-07-07 23:42 ` ellie
2024-07-08 1:27 ` rsbecker
2024-07-08 2:28 ` ellie
2024-07-08 12:30 ` rsbecker
2024-07-08 12:41 ` ellie
2024-07-08 14:32 ` Konstantin Khomoutov
2024-07-08 15:02 ` rsbecker
2024-07-08 15:14 ` ellie
2024-07-08 15:31 ` rsbecker
2024-07-08 15:48 ` ellie
2024-07-08 16:23 ` rsbecker
2024-07-08 17:06 ` ellie
2024-07-08 17:38 ` rsbecker [this message]
2024-07-08 16:09 ` Emanuel Czirai
2024-07-08 15:44 ` Konstantin Khomoutov
2024-07-08 16:27 ` rsbecker
2024-07-14 12:00 ` ellie
2024-07-24 6:42 ` ellie
2025-09-08 2:34 ` Ellie
2024-09-30 21:01 ` Ellie
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='003d01dad15d$abd3f750$037be5f0$@nexbridge.com' \
--to=rsbecker@nexbridge$(echo .)com \
--cc=el@horse64$(echo .)org \
--cc=git@vger$(echo .)kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox