Question

I’m trying to find a linear classifier that minimises training error (0/1 loss), in cases where the data is non-linearly separable. Specifically, I’m looking for a way of determining weights:

sign(weights' * features) = label

for features \in {0, 1}^d, label \in {-1, 1} and real-valued weights. I have N training instances, and I want the above equation to hold for the maximum possible number of instances. I know something like a hard-margin SVM would work if the problem was always separable, but I also need to find a solution when it is not.

(This task may sound a bit esoteric, but please don’t advise me on what to do instead of looking for a minimum-training-error linear classifier – what I have described is definitely the problem that I want to solve!)

Answer 1

Matlab includes a non-linear SVM toolbox that’s really easy to use.
Have a look at svmtrain and svmclassify.
To select the kernel function you have to use the 'kernel_function' argument, although the default is 'linear' which is what you’re trying to do.
The 'boxconstraint' argument let’s you select C which is the parameter for the soft constraint.

Edit

I have found this paper which describes a method to minimize the 0/1 loss.