How would you go about parsing a string of free form text to detect things like locations and names based on a dictionary of location and names? In my particular application there will be tens of thousands if not more entries in my dictionaries so I’m pretty sure just running through them all is out of the question. Also, is there any way to add “fuzzy” matching so that you can also detect substrings that are within x edits of a dictionary word? If I’m not mistaken this falls within the field of natural language processing and more specifically named entity recognition (NER); however, my attempt to find information about the algorithms and processes behind NER have come up empty. I’d prefer to use Python for this as I’m most familiar with that although I’m open to looking at other solutions.
How would you go about parsing a string of free form text to detect
Share
You might try downloading the Stanford Named Entity Recognizer:
http://nlp.stanford.edu/software/CRF-NER.shtml
If you don’t want to use someone else’s code and you want to do it yourself, I’d suggest taking a look at the algorithm in their associated paper, because the Conditional Random Field model that they use for this has become a fairly common approach to NER.
I’m not sure exactly how to answer the second part of your question on looking for substrings without more details. You could modify the Stanford program, or you could use a part-of-speech tagger to mark proper nouns in the text. That wouldn’t distinguish locations from names, but it would make it very simple to find words that are x words away from each proper noun.