Question

I am working on a project that deals with analyzing a very large amount of data, so I discovered MapReduce fairly recently, and before i dive any further into it, i would like to make sure my expectations are correct.

The interaction with the data will happen from a web interface, so response time is critical here, i am thinking a 10-15 second limit. Assuming my data will be loaded into a distributed file system before i perform any analysis on it, what kind of a performance can i expect from it?

Let’s say I need to filter a simple 5GB XML file that is well formed, has a fairly flat data structure and 10,000,000 records in it. And let’s say the output will result in 100,000 records. Is 10 seconds possible?

If it, what kind of hardware am i looking at?
If not, why not?

I put the example down, but now wish that I didn’t. 5GB was just a sample that i was talking about, and in reality I would be dealing with a lot of data. 5GB might be data for one hour of the day, and I might want to identify all the records that meet a certain criteria.

A database is really not an option for me. What i wanted to find out is what is the fastest performance i can expect out of using MapReduce. Is it always in minutes or hours? Is it never seconds?

Answer 1

MapReduce is good for scaling the processing of large datasets, but it is not intended to be responsive. In the Hadoop implementation, for instance, the overhead of startup usually takes a couple of minutes alone. The idea here is to take a processing job that would take days and bring it down to the order of hours, or hours to minutes, etc. But you would not start a new job in response to a web request and expect it to finish in time to respond.

To touch on why this is the case, consider the way MapReduce works (general, high-level overview):

• A bunch of nodes receive portions of
  the input data (called splits) and do
  some processing (the map step)

• The intermediate data (output from
  the last step) is repartitioned such
  that data with like keys ends up
  together. This usually requires some
  data transfer between nodes.

• The reduce nodes (which are not
  necessarily distinct from the mapper
  nodes – a single machine can do
  multiple jobs in succession) perform
  the reduce step.

• Result data is collected and merged
  to produce the final output set.

While Hadoop, et al try to keep data locality as high as possible, there is still a fair amount of shuffling around that occurs during processing. This alone should preclude you from backing a responsive web interface with a distributed MapReduce implementation.

Edit: as Jan Jongboom pointed out, MapReduce is very good for preprocessing data such that web queries can be fast BECAUSE they don’t need to engage in processing. Consider the famous example of creating an inverted index from a large set of webpages.