Question

As stated in the title, I’m simply looking for algorithms or solutions one might use to take in the twitter firehose (or a portion of it) and

a) identify questions in general
b) for a question, identify questions that could be the same, with some degree of confidence

Thanks!

Answer 1

(A)

I would try to identify questions using machine learning and the Bag of Words model.

1. Create a labeled set of twits, and label each of them with a binary
   flag: question or not question.
2. Extract the features from the training set. The features are traditionally words, but at least for any time I tried it – using bi-grams significantly improved the results. (3-grams were not helpful for my cases).
3. Build a classifier from the data. I usually found out SVM gives better performance then other classifiers, but you can use others as well – such as Naive Bayes or KNN (But you will probably need feature selection algorithm for these).
4. Now you can use your classifier to classify a tweet.¹

(B)

This issue is referred in the world of Information-Retrieval as "duplicate detection" or "near-duplicate detection".

You can at least find questions which are very similar to each other using Semantic Interpretation, as described by Markovitch and Gabrilovich in their wonderful article Wikipedia-based Semantic Interpretation for Natural Language Processing. At the very least, it will help you identify if two questions are discussing the same issues (even though not identical).

The idea goes like this:

1. Use wikipedia to build a vector that represents its semantics, for a term t, the entry vector_t[i] is the tf-idf score of the term i as it co-appeared with the term t. The idea is described in details in the article. Reading the 3-4 first pages are enough to understand it. No need to read it all.²
2. For each tweet, construct a vector which is a function of the vectors of its terms. Compare between two vectors – and you can identify if two questions are discussing the same issues.

EDIT:

On 2nd thought, the BoW model is not a good fit here, since it ignores the position of terms. However, I believe if you add NLP processing for extracting feature (for examples, for each term, also denote if it is pre-subject or post-subject, and this was determined using NLP procssing), combining with Machine Learning will yield pretty good results.

---

(1) For evaluation of your classifier, you can use cross-validation, and check the expected accuracy.

(2) I know Evgeny Gabrilovich published the implemented algorithm they created as an open source project, just need to look for it.