Question

Given data in the following format (tag_uri image_uri image_uri image_uri …), I need to turn them into Hadoop SequenceFile format for further processing by Mahout (e.g. clustering)

http://flickr.com/photos/tags/100commentgroup http://flickr.com/photos/34254318@N06/4019040356 http://flickr.com/photos/46857830@N03/5651576112
http://flickr.com/photos/tags/100faves http://flickr.com/photos/21207178@N07/5441742937
...

Before this I would turn the input into csv (or arff) as follows

http://flickr.com/photos/tags/100commentgroup,http://flickr.com/photos/tags/100faves,...
0,1,...
1,1,...
...

with each row describes one tag. Then the arff file is converted into a vector file used by mahout for further processing. I am trying to skip the arff generation part, and generate a sequenceFile instead. If I am not mistaken, to represent my data as a sequenceFile, I would need to store each row of the data with $tag_uri as key, then $image_vector as value. What is the proper way of doing this (if possible, can I have the tag_url for each row to be included in the sequencefile somewhere)?

Some references that I found, but not sure if they are relevant:

1. Writing a SequenceFile
2. Formatting input matrix for svd matrix factorization (can I store my matrix in this form?)
3. RandomAccessSparseVector (considering I only list images that are assigned with a given tag instead of all the images in a line, is it possible to represent it using this vector?)
4. SequenceFile write
5. SequenceFile explanation

Answer 1

You just need a SequenceFile.Writer, which is explained in your link #4. This lets you write key-value pairs to the file. What the key and value are depends on your use case, of course. It’s not at all the same for clustering versus matrix decomposition versus collaborative filtering. There’s not one SequenceFile format.

Chances are that the key or value will be a Mahout Vector. The thing that knows how to write a Vector is VectorWritable. This is the class you would use to wrap a Vector and write it with SequenceFile.Writer.

You would need to look at the job that will consume it to make sure you’re passing what it expects. For clustering, for example, I think the key is ignored and the value is a Vector.