Progression not functional once a single error / fatal line was encountered

To run the script properly, just create a creds.py file with a token variable, and a .tmp folder

import creds

from git import Repo, RemoteProgress
from pathlib import Path

import uuid


username = "gitpython-developers"
repo_name = "GitPython"
uuid = uuid.uuid4().hex[:20]

# ---------------

class Progress(RemoteProgress):
    def update(self, op_code, cur_count, max_count=None, message=''):
        print(f"{op_code} {cur_count} {max_count} {message}")

repo_url = f'https://{creds.token}:x-oauth-basic@github.com/{username}/{repo_name}'
repo_path = Path('.tmp') / Path(uuid)
print(f'Cloning the commits history of {repo_name}...')
err = None
try:
    Repo.clone_from("https://github.com/gitpython-developers/GitPython", repo_path, progress=Progress(), multi_options=["--filter=tree:0"])
    #Repo.clone_from(repo_url, repo_path, progress=Progress(), multi_options=["--filter=tree:0"])
except Exception as error:
    err = error
    print(err)

If you comment the 1st Repo.clone_from and uncomment the 2nd, the progress will not work anymore, even though the progress is showing up using the git binary.

Thanks a lot for putting in the time to make this issue reproducible. With the latest GitPython from main and tested with my own token, the following code produced progress messages just like the one above.

import creds

from git import Repo, RemoteProgress
from pathlib import Path

import uuid


username = "gitpython-developers"
repo_name = "GitPython"
uuid = uuid.uuid4().hex[:20]

# ---------------

class Progress(RemoteProgress):
    def update(self, op_code, cur_count, max_count=None, message=''):
        print(f"{op_code} {cur_count} {max_count} {message}")

repo_url = f'https://{creds.token}:x-oauth-basic@github.com/{username}/{repo_name}'
repo_path = Path('.tmp') / Path(uuid)
print(f'Cloning the commits history of {repo_name}...')
err = None
try:
    #Repo.clone_from("https://github.com/gitpython-developers/GitPython", repo_path, progress=Progress(), multi_options=["--filter=tree:0"])
    Repo.clone_from(repo_url, repo_path, progress=Progress(), multi_options=["--filter=tree:0"])
except Exception as error:
    err = error
    print(err)

Thus I was unable to reproduce the issue, running on python 3.8 on MacOS. This is something I would expect as there is no special case for the URL at all.

GitPython/git/repo/base.py

Lines 1012 to 1016 in 96f8f17

    
           proc = git.clone(multi, Git.polish_url(url), clone_path, with_extended_output=True, as_process=True, 
        
                            v=True, universal_newlines=True, **add_progress(kwargs, git, progress)) 
        
           if progress: 
        
               handle_process_output(proc, None, to_progress_instance(progress).new_message_handler(), 
        
                                     finalize_process, decode_streams=False)

Maybe there is something else that influences whether or not progress is sent that isn't present in my particular setup?

@Byron

Thanks for your answer.
I updated Git SCM and GitPython to the latest version, but I can't manage to get it working. Maybe it's because of Windows ? But it would be no sense since we get the same progress with the two different URLs using the Git CLI.
Do you know a GitPython contributor on Windows that can test the same thing ? @Byron

Maybe it's the way you get the stream that doesn't work the same way on Windows.
I'll try to play with GitPython and put some print() everywhere.

Okay it works well on my Linux VM.

So the problem should be Windows

Thanks for sorting this out, very helpful! It's definitely something the next major release of GitPython will fix.
In the mean time, you could call git directly, using subprocess, and have its stderr/progress written directly to the terminal if that's an option. Parsing it yourself is what GitPython is trying to do, which apparently is an issue on windows at least the way it's implemented currently.
If you come across a fix for this it would be great to read about it here.
Thank you.

I'll let you know, thank you, for the moment I'm trying to understand all your pumps / streams things aha 😄
By the way, were you able to confirm this case doesn't work for another Windows user ? Just to be sure that it's not because of my environment

I added a print in this function, before the two handlers :

GitPython/git/cmd.py

Lines 78 to 89 in 96f8f17

    
           def pump_stream(cmdline, name, stream, is_decode, handler): 
        
               try: 
        
                   for line in stream: 
        
                       if handler: 
        
                           if is_decode: 
        
                               line = line.decode(defenc) 
        
                           handler(line) 
        
               except Exception as ex: 
        
                   log.error("Pumping %r of cmd(%s) failed due to: %r", name, remove_password_if_present(cmdline), ex) 
        
                   raise CommandError(['<%s-pump>' % name] + remove_password_if_present(cmdline), ex) from ex 
        
               finally: 
        
                   stream.close()

And the stream is well received in the two case, here is the x-oauth-basic case :

And with a normal link :

So the issue should be in the output processing.

I can't confirm it nor deny it. My spotty memory tells me this isn't the first time I hear about an issue like it though, if that helps.
Awesome, I have a feeling you will soon have a PR ready which fixes this issue once and for all.

I finally figured out the issue !

GitPython printed this line at the end so I didn't think it was the issue, but it was : "fatal: credential-cache unavailable; no unix socket support" and the stream got it at the beginning.
And since in your parsing function here :

GitPython/git/util.py

Lines 426 to 446 in 96f8f17

    
               def _parse_progress_line(self, line: AnyStr) -> None: 
        
                   """Parse progress information from the given line as retrieved by git-push 
        
                   or git-fetch. 
        
                   - Lines that do not contain progress info are stored in :attr:`other_lines`. 
        
                   - Lines that seem to contain an error (i.e. start with error: or fatal:) are stored 
        
                   in :attr:`error_lines`.""" 
        
                   # handle 
        
                   # Counting objects: 4, done. 
        
                   # Compressing objects:  50% (1/2) 
        
                   # Compressing objects: 100% (2/2) 
        
                   # Compressing objects: 100% (2/2), done. 
        
                   if isinstance(line, bytes):   # mypy argues about ternary assignment 
        
                       line_str = line.decode('utf-8') 
        
                   else: 
        
                       line_str = line 
        
                   self._cur_line = line_str 
        
                   if self.error_lines or self._cur_line.startswith(('error:', 'fatal:')): 
        
                       self.error_lines.append(self._cur_line) 
        
                       return

if there is only one line that starts with "error:" or "fatal:", it put all the next output in the "self.error_lines" variable even if they are not errors, and so that condition still True until the end.
My solution was to fix the error itself by doing it :

git config --global credential.helper wincred

But what do you think about editing the parsing function to let the progress going on even if there was an error at the beginning ?

Fantastic! I think it shouldn’t consider everything else an error just beca It found one. Maybe worth a try removing that check and see what breaks in CI. A PR is definitely welcome.

To reactivate this error case, we can do this command : git config --global credential.helper cache
Maybe it works depending on how you manage your credentials on your system / if you have Windows, idk

I submitted a PR to avoid this issue ! #1248

Navigation Menu

Repository navigation

Progression not functional once a single error / fatal line was encountered #1233

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	proc = git.clone(multi, Git.polish_url(url), clone_path, with_extended_output=True, as_process=True,
	v=True, universal_newlines=True, **add_progress(kwargs, git, progress))
	if progress:
	handle_process_output(proc, None, to_progress_instance(progress).new_message_handler(),
	finalize_process, decode_streams=False)

	def pump_stream(cmdline, name, stream, is_decode, handler):
	try:
	for line in stream:
	if handler:
	if is_decode:
	line = line.decode(defenc)
	handler(line)
	except Exception as ex:
	log.error("Pumping %r of cmd(%s) failed due to: %r", name, remove_password_if_present(cmdline), ex)
	raise CommandError(['<%s-pump>' % name] + remove_password_if_present(cmdline), ex) from ex
	finally:
	stream.close()

	def _parse_progress_line(self, line: AnyStr) -> None:
	"""Parse progress information from the given line as retrieved by git-push
	or git-fetch.

	- Lines that do not contain progress info are stored in :attr:`other_lines`.
	- Lines that seem to contain an error (i.e. start with error: or fatal:) are stored
	in :attr:`error_lines`."""
	# handle
	# Counting objects: 4, done.
	# Compressing objects: 50% (1/2)
	# Compressing objects: 100% (2/2)
	# Compressing objects: 100% (2/2), done.
	if isinstance(line, bytes): # mypy argues about ternary assignment
	line_str = line.decode('utf-8')
	else:
	line_str = line
	self._cur_line = line_str

	if self.error_lines or self._cur_line.startswith(('error:', 'fatal:')):
	self.error_lines.append(self._cur_line)
	return

Progression not functional once a single error / fatal line was encountered #1233

Description

Activity

Byron commented on May 12, 2021

mxrch commented on May 20, 2021

mxrch commented on May 20, 2021

mxrch commented on May 20, 2021

Byron commented on May 20, 2021

mxrch commented on May 20, 2021

mxrch commented on May 20, 2021

Byron commented on May 20, 2021

mxrch commented on May 21, 2021

Byron commented on May 21, 2021

mxrch commented on May 21, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions