Question

I have a script that reads a large file line by line. The record separator ($/) that I would like to use is (\n). The only problem is that the data on each line contains CRLF characters (\r\n), which the program should not be considered the end of a line.

For example, here is a sample data file (with the newlines and CRLFs written out):

line1contents\n
line2contents\n
line3\r\ncontents\n
line4contents\n

If I set $/ = "\n", then it splits the third line into two lines. Ideally, I could just set $/ to a regex that matches \n and not \r\n, but I don’t think that’s possible. Another possibility is to read in the whole file, then use the split function to split on said regex. The only problem is that the file is too large to load into memory.

Any suggestions?

Answer 1

For this particular task, it sounds pretty straightforward to check your line ending and append the next line as necessary:

$/ = "\n";
...
while(<$input>) {
    while( substr($_,-2) eq "\r\n" ) {
        $_ .= <$input>;
    }
    ...
}

This is the same logic used to support line continuation in a number of different programming contexts.

You are right that you can’t set $/ to a regular expression.