Question

I find this question a little tricky. Maybe someone knows an approach to answer this question. Imagine that you have a dataset(training data) which you don’t know what it is about. Which features of training data would you look at in order to infer classification algorithm to classify this data? Can we say anything whether we should use a non-linear or linear classification algorithm?

By the way, I am using WEKA to analyze the data.

Any suggestions?
Thank you.

Answer 1

Wow, so you have some training data and you don’t know whether you are looking at features representing words in a document, or genese in a cell and need to tune a classifier. Well, since you don’t have any semantic information, you are going to have to do this soley by looking at statistical properties of the data sets.

First, to formulate the problem, this is more than just linear vs non-linear. If you are really looking to classify this data, what you really need to do is to select a kernel function for the classifier which may be linear, or non-linear (gaussian, polynomial, hyperbolic, etc. In addition each kernel function may take one or more parameters that would need to be set. Determining an optimal kernel function and parameter set for a given classification problem is not really a solved problem, there are only useful heuristics and if you google ‘selecting a kernel function’ or ‘choose kernel function’, you will be treated to many research papers proposing and testing various approaches. While there are many approaches, one of the most basic and well travelled is to do a gradient descent on the parameters– basically you try a kernel method and a parameter set , train on half your data points and see how you do. Then you try a different set of parameters and see how you do. You move the parameters in the direction of best improvement in accuracy until you get satisfactory results.

If you don’t need to go through all this complexity to find a good kernel function, and simply want an answer to linear or non-linear. then the question mainly comes down to two things: Non linear classifiers will have a higher risk of overfitting (undergeneralizing) since they have more dimensions of freedom. They can suffer from the classifier merely memorizing sets of good data points, rather than coming up with a good generalization. On the other hand a linear classifier has less freedom to fit, and in the case of data that is not linearly seperable, will fail to find a good decision function and suffer from high error rates.

Unfortunately, I don’t know a better mathematical solution to answer the question “is this data linearly seperable” other than to just try the classifier itself and see how it performs. For that you are going to need a smarter answer than mine.

Edit: This research paper describes an algorithm which looks like it should be able to determine how close a given data set comes to being linearly seperable.

http://www2.ift.ulaval.ca/~mmarchand/publications/wcnn93aa.pdf