Question

When running svndumpfilter2 on Windows, I get a problem that seems to take its origin in the fact that the dump file has sometimes CRLF endings.

Some files in the SVN database had CRLF line endings. But it seems that Python counts CRLF as one character (not counting the CR character as separate from the following LF in the content of the files). Thus, it fails read the right amount of characters, and misses the start of the next lump.

So my question is: how to tell Python to treat CRLF as two separate characters?

The stream is read from sys.stdin so I’m looking for a way to change the newline property of stdin. What is the “one right way” to do that in Python?

Answer 1

Update: One way that occurs to me is to explicitly set the mode of stdin to binary. So something like the following will read CRLF as two characters:

import msvcrt, os, sys

msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)
while True:
    ch = sys.stdin.read(1)
    print ord(ch)  # CRLF should appear as 13 followed by 10

Another way is to start python with the -u flag which results in a unbuffered stdin (as well as stdout and stderr). So just python -u myscript.py where myscript.py calls stdin.read(1) with no other changes. See python --help for more information on this.

Old: If you’re on windows, Python should be able to handle this without any intervention when you call sys.stdin.readline (or simply iterate over sys.stdin which is a file like object). Are you using sys.stdin.read instead? If so, you need to handle that case yourself.