Question

I don’t understand what types of apps can be used with Hadoop. Does each task have to be tailored for hadoop/mapreduce. For example, can you just associate any long running java processed with it? Or do you specifically have to tailor your app/task for hadoop. I guess a good example would be using lucene and hadoop for indexing.

Answer 1

MapReduce is a processing model; it tells you exactly what your processing task should fit into.

1. Your processing must be batch oriented
2. You must be able to transform your work into a (set of) map and reduce steps.
3. In order to have any advantage of the scalability properties of MapReduce you must be able to split the work into enough independent (!!) pieces that can be processed separately.

Hadoop does (among other things) MapReduce with the added advantage that you can actually run a job reliably on 1000 systems in parallel (if you have enough independent pieces).

Given those constraints: some things cannot be done and a lot of things can be done.
Analyzing logfiles (i.e. a large set of independent lines) or even webanalytics (every a single visitor/session did can be processed separately) are amongst the most common applications.

So yes, your task must be transformed to fit in the model for it to work.