Question

I am implementing K nearest neighbour algorithm for a very sparse data. I want to calculate the distance between a test instance and each sample in the training set, but I am confused.

Because most of the features in training samples don’t exist in test instance or vice versa (missing features).

How can I compute the distance in this situation?

Answer 1

To make sure I’m understanding the problem correctly: each sample forms a very sparsely filled vector. The missing data is different between samples, so it’s hard to use any Euclidean or other distance metric to gauge similarity of samples.

If that is the scenario, I have seen this problem show up before in machine learning – in the Netflix prize contest, but not specifically applied to KNN. The scenario there was quite similar: each user profile had ratings for some movies, but almost no user had seen all 17,000 movies. The average user profile was quite sparse.

Different folks had different ways of solving the problem, but the way I remember was that they plugged in dummy values for the missing values, usually the mean of the particular value across all samples with data. Then they used Euclidean distance, etc. as normal. You can probably still find discussions surrounding this missing value problem on that forums. This was a particularly common problem for those trying to implement singular value decomposition, which became quite popular and so was discussed quite a bit if I remember right.

You may wish to start here:
http://www.netflixprize.com//community/viewtopic.php?id=1283

You’re going to have to dig for a bit. Simon Funk had a little different approach to this, but it was more specific to SVDs. You can find it here: http://www.netflixprize.com//community/viewtopic.php?id=1283
He calls them blank spaces if you want to skip to the relevant sections.

Good luck!