Question

I need to convert a bunch of files to utf-8 in Python, and I have trouble with the ‘converting the file’ part.

I’d like to do the equivalent of:

iconv -t utf-8 $file > converted/$file # this is shell code 

Thanks!

Answer 1

You can use the codecs module, like this:

import codecs BLOCKSIZE = 1048576 # or some other, desired size in bytes with codecs.open(sourceFileName, 'r', 'your-source-encoding') as sourceFile:     with codecs.open(targetFileName, 'w', 'utf-8') as targetFile:         while True:             contents = sourceFile.read(BLOCKSIZE)             if not contents:                 break             targetFile.write(contents) 

EDIT: added BLOCKSIZE parameter to control file chunk size.