I have a classification problem where I would like to predict an outcome, but would like my classifier to get several ‘attempts’ at the answer (something like placing an each-way bet), rather than a single classification which is either correct or incorrect, and was wondering about the best process for this.
Example: Given outcomes A, B, C, and D, I would like to predict that it will be ‘A or B’, or ‘A or C’, and the ‘correct’ solution(s) (those that at least contain the right individual answer) affect the learning process accordingly.
So far, my thoughts have been to split the data set up into bins, more or less as above (A or C) and train a classifier in the usual way, or to train multiple classifiers such that they are diverse, and simply combine the results, but I was wondering if there is a better/Different way? I’m sure this can’t be a unique problem, but I’m not sure of the correct terminology to Google.
I don’t know if it’s a related problem, but is there also a way to include in the options ‘I don’t know’ – ie. don’t make a classification?
A lot of classifiers can do what you want.
Naive Bayes can give you probabilities for each label, so you can take the k most probable labels instead of just the single most probable label and output that.
Logistic Regression, SVMs can also give you a score for each label, letting you do something similar.
Another trick is to slightly perturb the input feature vector and feed it to the classifier. Repeat that several times, and you get not one output label, but several. You can count and sort them by frequency to get multiple potential answers. You can then make some cutoff criteria to pick only a subset of those labels and return them to the user.