Question

(Problem solved, please see the updates)

I have some files that have incorrect filenames because of encoding issues. So I want to write a python script to process it. However, I encounter a strange problem.

To better illustrate, I will use an example: the file name is displayed as ¹þÀï·ÑÇ.mp3.
However, the following result is different:

# only one mp3 file is in this directory:
$ ls *mp3  | hexdump 
0000000 c2 b9 c3 be 41 cc 80 69 cc 88 41 cc 82 c2 b7 4e
0000010 cc 83 43 cc a7 2e 6d 70 33 0a                  
000001a

$ echo "¹þÀï·??Ç.mp3"  | hexdump 
0000000 c2 b9 c3 be c3 80 c3 af c3 82 c2 b7 c3 91 c3 87
0000010 2e 6d 70 33 0a                                 
0000015

Basically, the second string (or bytes) is the one I wanted, but in my Python script, the command line arguments always give me the first string. I have no way to get around.
I notice that this only happens in Mac OS X. Hence, I suspect the argument is somehow encoded or processed by the bash/system/python.
Here is a listing of my tools:

• Python: 2.7.2
• OS: Mac OS X 10.6.7
• Shell: GNU bash, version 3.2.48(1)-release (x86_64-apple-darwin10.0)

Update: the following code works well in my Arch Linux, but suffers from the above problem in my Mac OS X:

#!/usr/bin/env python

import sys 
import os
for name in sys.argv[1:]:
    try:
        # This line does the magic:
        new_name = name.decode('utf8').encode('latin-1').decode('gbk')
        new_name_utf8 = new_name.encode('utf8')
        if name != new_name_utf8:
            print "%s -> %s" % (name, new_name_utf8)
            os.rename(name, new_name)
    except:
        print "Ignoring %s" % name

In shell, run:

$ ./the_script *mp3 # Let bash pass the file name string

You can run the above code for the string ¹þÀï·ÑÇ.mp3, and it should be correctly identified as 哈里路亚.mp3. Note that you must have a UTF-8 locale and a correct Chinese font that supports Unicode to display it correctly, or check the following image:

FYI: the GBK encoded filename does not recognize by my download program, and it is interpreted as a unicode string, which is encoded as UTF-8. The non-ascii byte in the original file is interpreted as a code point of Unicode and encoded using UTF-8, which causes the problem.

Update2: The script that are portable between Mac and Linux is now uploaded here.

Answer 1

The problem is that MacOS X’s default filesystem changes all filenames you give it to an unusual normalization form which does not use precomposed characters. The unicodedata Python module allows converting between these forms, for example:

import unicodedata
print len(unicodedata.normalize("NFD", u"\u00C7"))
print len(unicodedata.normalize("NFC", u"\u00C7"))

These print 2 and 1, respectively.