Question

I have an app that reads large customer-supplied data files. It works perfectly with several but, on one file I received today, it is failing with:

ArgumentError: invalid byte sequence in UTF-8

I am using String.match to look for regex patterns.

When I look at the file, nothing seems different from the ones that work.

Advice?

Edit: it looks like there there is an ‘xE9’ character in a user name.

Answer 1

Thanks to @muistooshort ‘s help, I opened the file in ISO mode and then, reading line by line, convert to UTF-8.

myfile = File.open( 'thefile.txt', 'r:iso8859-1' )
  while rawline = myfile.gets
  line = rawline.force_encoding( 'utf-8' )
  # proceed...
end