Question

I’m new to Natural Language Processing and I’m a confused about the terms used.

What is tokenization? POS tagging? Entity Identify?

Tokenization is only split the text in parts that can have a meaning or give a meaning for these parts? And the meaning, what is the name when I determine that something is a noun, verb or adjetive. And if I want to divide into dates, names, currency?

I need a simple explanation about the areas/terms used in NLP.

Answer 1

To add to dmn’s explanation:

In general, there are two themes you should care about in NLP:

1. Statistical vs Rule-Based Analysis

2. Lightweight vs Heavyweight Analysis

Statistical Analysis uses statistics machine learning techniques to classify text and in general have good precision and good recall. Rule-Based Analysis techniques basically use hand-built rules and have very good precision but terrible recall (basically they identify the cases in your rules, but nothing else).

Lightweight vs Heavyweight Analysis are the two approaches you’ll see in the field. In general, academic work is heavyweight, featuring parsers, fancy classifiers and lots of very high tech NLP stuff. In industry, by and large the focus is on data, and a lot of the academic stuff scales poorly and going beyond standard statistical or machine learning techniques doesn’t bring you much. For example, parsing is largely useless (and slow) and as such keyword and ngram analysis is actually pretty useful, especially when you have a lot of data. For example, Google Translate isn’t apparently that fancy behind the scenes- they just have so much data they can crush everybody else no matter how refined their translation software is.

The upshot of this is in industry there’s a lot of machine learning and math, but the NLP stuff is used is not very sophisticated, because the sophisticated stuff really doesn’t work well. Far preferred is using user data like clicks on related subjects and mechanical turk… and this works very well as people are far better at understanding natural language than computers.

Parsing is break a sentence down into phrases, say verb phrase, noun phrase, prepositional phrase, etc and get a grammatical tree. You can use the online version of the Stanford Parser to play with examples and get a feel for what a parser does. For example, Let’s say we have the sentence

My cat's name is Pat.

Then we do POS tagging:

My/PRP$ cat/NN 's/POS name/NN is/VBZ Pat/NNP ./.

Using the POS tags and a trained statistical parser, we get a parse tree:

(ROOT
  (S
    (NP
      (NP (PRP$ My) (NN cat) (POS 's))
      (NN name))
    (VP (VBZ is)
      (NP (NNP Pat)))
    (. .)))

We can also do a slightly different type of parse called a dependency parse:

poss(cat-2, My-1)
poss(name-4, cat-2)
possessive(cat-2, 's-3)
nsubj(Pat-6, name-4)
cop(Pat-6, is-5)

N-Grams are basically sets of adjacent words of length n. You can look at n-grams in Google’s data here. You can also do character n-grams which are used heavily for spelling correction.

Sentiment Analysis is analyzing text to extract how people feel about something or in what light things (such as brands) are mentioned. This involves a lot of looking at words that denote emotion.

Semantic Analysis is analyzing the meaning of text. Often this takes the form of taxonomies and ontologies where you group concepts together (dog,cat belong to animal and pet) but it is a very undeveloped field. Resources like WordNet and Framenet are useful here.