I am looking for word alignment tools and algorithms.
I am dealing with bilingual English – Hindi text, and currently working on
Could you please suggest any other algorithm/tool which is language independent and which could achieve Statistical word alignment for parallel English Hindi Corpora and its evaluation.
Some tools are best for certain languages; could you please tell me how true that is and, if so, could you please provide an example of what would be better suited for Asian languages like Hindi. Counter-examples of what one shouldn’t I use for such languages is also welcome.
I have heard a bit about Uplug word aligner… Could someone tell me if this tool is useful for my purpose.
Thank you.. 🙂
The Berkeley Aligner is very good. By doing joint training of the IBM word alignment models, it’s able to get a much lower alignment error rate (AER) than older packages like GIZA++.
It also supports some more advanced features such as syntactic distortion (i.e., using parse tree information to get better alignments). For this, you’ll only need parse trees for one of the language pairs. So, you should be okay doing Hindi<->English, since there are plenty of freely available and good English parsers.
If you decide not to go with the Berkeley Aligner, you should probably just use GIZA++. For years, it has been essentially the standard word aligner in the machine translation community.