I am working on a research and I would like to come up with a method that refuses to classify some constant portion of test data (e.g. 20%, one in five classifications can be answered as “i am not sure” by the algorithm). The idea is to have an algorithm that can effectively choose which classifications are the most probable to be false and refuse to answer them (in order to improve overal accuracy).
I wonder if there is any general machine learning method (indepenedent of classifier used) to achieve this?
Any answer will help, thanks.
A logistic regression classifier will output the probability that an example belongs to a positive or negative category. Setting a threshold to marking an example “unknown” would work in this case. For example, anything that returned <.6 probability of either positive or negative could be marked as unknown.
Another approach, as another poster suggested, is to treat the problem as a ranking problem. Linear classifiers (like SVMs and logistic regression) output an example’s distance from a separating hyperplane. You can use the absolute value of this distance to rank examples then classify 20% of the test examples with the lowest rank (closest to the separating hyperplane) as unknown.