Question

What are common approaches for translating certain words (or expressions) inside a given text, when the text must be reconstructed (with punctuations and everythin.) ?

The translation comes from a lookup table, and covers words, collocations, and emoticons like L33t, CUL8R, :-), etc.

Simple string search-and-replace is not enough since it can replace part of longer words (cat > dog ≠> caterpillar > dogerpillar).

Assume the following input:

s = "dogbert, started a dilbert dilbertion proces cat-bert :-)"

after translation, i should receive something like:

result = “anna, started a george dilbertion process cat-bert smiley“

I can’t simply tokenize, since i loose punctuations and word positions.

Regular expressions, works for normal words, but don’t catch special expressions like the smiley 🙂 but it does .

re.sub(r'\bword\b','translation',s) ==> translation
re.sub(r'\b:-\)\b','smiley',s) ==> :-)

for now i’m using the above mentioned regex, and simple replace for the non-alphanumeric words, but it’s far from being bulletproof.

(p.s. i’m using python)

Answer 1

I had a similar problem with standard emoticons to be replaced with values. Here is a list of emoticons. I had them in a plain text file (so that I can append/delete to it as and when required) separated by tab like.

:[        -1
:/        -1
:(          -1
:)         1

Then read it into a dictionary

emoticons = {}          
for line in open('data/emoticons.txt').xreadlines():   
        symbol, value = line.split('\t')                                           
        emoticons[str(symbol)] = int(value)

Then a lookup function

def mark_emoticons(t):
    for w, v in emoticons.items():
        match = re.search(re.escape(w),t)
            if match:
                print w, "found "

Call the function with

mark_emoticons('Hello ladies! How are you? Fantastic :) Look at your man ...')

As for L33t-speak I have a separate file slangs.txt, which looks like

u   you
ur  you are
uw  you are welcome 
wb  welcome back 
wfm works for me 
wtf what the fuck

A similar function to read it to dictionary slangs{} and a similar function to replace the slangs.

def mark_slangs(t):        
    for w, v in slangs.items():
            s = r'\b' + w + r'\b'
            match = re.search(s,t)
            if match:
                    #print w, "found in:",t, "replacing with",readtable.slangs[w]
                    t = re.sub(w,slangs[w].rstrip(),t)
                    ...

From Python library the re.escape()

re.escape(string)
Return string with
all non-alphanumerics backslashed;
this is useful if you want to match an
arbitrary literal string that may have
regular expression metacharacters in
it.

Based on your needs you might want to use re.findall()