Question

I am writing a Python file that needs to read in several files of different types. I am reading the files in line by line with the traditional for line in f after using f = open("file.txt", "r").

This doesn’t seem to be working for all files. My guess is some files end with different encodings (such as \r\n versus just \r). I can read the whole file in and do a string split on \r, but that is hugely costly and I’d rather not. Is there a way to make the readline method of Python recognize both end-of-line variations?

Answer 1

Use the universal newline support — see http://docs.python.org/library/functions.html#open

In addition to the standard fopen()
values mode may be ‘U’ or ‘rU’. Python
is usually built with universal
newline support; supplying ‘U’ opens
the file as a text file, but lines may
be terminated by any of the following:
the Unix end-of-line convention ‘\n’,
the Macintosh convention ‘\r’, or the
Windows convention ‘\r\n’. All of
these external representations are
seen as ‘\n’ by the Python program. If
Python is built without universal
newline support a mode with ‘U’ is the
same as normal text mode. Note that
file objects so opened also have an
attribute called newlines which has a
value of None (if no newlines have yet
been seen), ‘\n’, ‘\r’, ‘\r\n’, or a
tuple containing all the newline types
seen.