Question

I’ve got a Perl program that I wrote on Windows. It starts with:

$unused_header = <STDIN>;
my @header_fields = split('\|\^\|', $unused_header, -1);

Which should split input that consists of a very large file of:

The|^|Quick|^|Brown|^|Fox|!|

Into:

{The, Quick, Brown, Fox|!|}

Note: This line just does the headre alone, theres another one like it to do the repetitive data lines.

It worked great on windows, but on linux it fails. However, if I define a string with the same contents within Perl, and run the split on that, it works fine.

I think it’s a UTF-16 encoding handling issue, but I’m not sure how to handle it. Does anyone know how I can get perl to understand the UTF-16 being piped into STDIN?

I found: http://www.haboogo.com/matching_patterns/2009/01/utf-16-processing-issue-in-perl.html but I’m not sure what to do with it.

Answer 1

If STDIN is UTF-16, use one of the following

binmode(STDIN, ':encoding(UTF-16le)');   # Byte order used by Windows.
binmode(STDIN, ':encoding(UTF-16be)');   # The other byte order.
binmode(STDIN, ':encoding(UTF-16)');     # Use BOM to determine byte order.