Fetching from GerritHub sometimes randomly fails [40013362]

In Progress

Bug

Host-Gerrithub

Status Update

No update yet.

Description

ma...@truewealth.ch

created issue #1

Jan 20, 2021 01:25PM

GerritHub/GitHub username: martynas.subonis@truewealth.ch

GerritHub/GitHub projects names: true-wealth/b2b

Full URL exhibiting the problem:

On some occasions (randomly) fetching from GerritHub fails with the following errors:

git fetch <...>  
ERROR: Error cloning remote repo 'gerrithub'  
hudson.plugins.git.GitException: Command "git fetch <...>" returned status code 128:  
stdout:   
stderr: Connection to review.gerrithub.io closed by remote host.  
fatal: The remote end hung up unexpectedly  
fatal: protocol error: bad pack header

(no "remote: Counting objects" or "remote: Finding sources" logs are displayed, and it is not a timeout issue as well from our side as well).

Expected behavior: The fetching should consistently succeed.

Observed behavior: The fetching sometimes fails with errors "fatal: The remote end hung up unexpectedly" and "fatal: protocol error: bad pack header".

Timestamp when the error occurred (include timezone):

The exact timestamp of some of these failures would be:

January 13th, 16:04:34 CEST  
January 14th, 14:12:53 CEST

Has this worked before? If yes, when?

For us, this seems to have been a recurring issue over a span of few months (happens once in 2-3 days).

If using git command-line: output of "which git && git --version"

which git # /usr/bin/git

git --version # git version 2.17.1

Comments

lu...@gmail.com <lu...@gmail.com> #2Jan 20, 2021 03:58PM

Which protocol are you using from Jenkins?

ma...@truewealth.ch <ma...@truewealth.ch> #3Jan 21, 2021 06:32AM

The git fetch is being done via ssh. The ssh version that is being used on our Jenkins Workers is:

ssh -V
OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017

ma...@truewealth.ch <ma...@truewealth.ch> #4Feb 17, 2021 10:17AM

Hello. Was there any progress or follow-up regarding this issue?

lu...@gmail.com <lu...@gmail.com> #5Feb 17, 2021 11:08AM

The error "fatal: protocol error: bad pack header" tells me that isn't something related to Gerrit Code Review.

Can you confirm that is your source IP? What is the repository name you are cloning?
Does it currently happen? How often? Can you provide more recent timestamps?

Luca.

lu...@gmail.com <lu...@gmail.com> #6Feb 17, 2021 11:14AM

I found something:

On the 13th of January:
[2021-01-13T15:04:34.815Z] 48d45da5 [SshCommandDestroy-0] true-wealth a/1009846 git-upload-pack./true-wealth/b2b 2ms 354790ms killed

On the 14th of January:
[2021-01-14T13:12:53.108Z] 02930af6 [SshCommandDestroy-0] true-wealth a/1009846 git-upload-pack./true-wealth/b2b 2ms 355998ms killed

Basically, it looks like the operation is taking more than 5 minutes and therefore is killed.

lu...@gmail.com <lu...@gmail.com> #7Feb 17, 2021 01:12PM

It looks like that the

https://crbug.com/gerrit/14097 has caused the "Run GC" button to disappear and, since then, you guys haven't done GC of the repo, causing a sluggish performance that ended up eventually in fetches timing out.

I believe Marcin is working on fixing

https://crbug.com/gerrit/14097, so that you guys can then start again doing GC by yourselves.

lu...@gmail.com <lu...@gmail.com> #8Feb 17, 2021 01:12PM

Accepted by lu...@gmail.com.

[Empty comment from Monorail migration]

ma...@truewealth.ch <ma...@truewealth.ch> #9Feb 22, 2021 08:43AM

Big thanks for looking into it.

do...@gmail.com <do...@gmail.com> #10Jun 23, 2021 06:31AM

Vy kurvy

ek...@google.com <ek...@google.com> #11Jun 29, 2022 02:09PM

[Empty comment from Monorail migration]

ek...@google.com <ek...@google.com> #12Jul 13, 2022 08:43AM

[Monorail components: Hosting>gerrithub]

is...@google.com <is...@google.com> #13Jul 13, 2022 08:43AM

Edits were made to reflect the following in Monorail: auto-CCs.