Is this the only way to achieve scale with map & reduce?
What I hate about this is that persistence ignorance is all jacked up. It should be that HDFS expose itself via an interface, you shouldn’t care about what language it’s written on, or why. Just like how you can write ODBC and it’ll plug into Oracle, Sql Server and what not running on any operating system.
I am aware of Hive, but I do not think it is well suited for heavier computation such as matrix manipulation, Gaussian analysis, etc.
The other problem is writing complex instruction set and the dependency such thing entails. It means you have to figure out how to port your code and install it along with any dependencies onto the server itself. This is a lot of infrastructural cost! And it’s also difficult to do inside a (Platform as a service) Paas cloud.
An example using Hadoop streaming for instance. You gotta make sure your binary is compiled against the target server kernel. eg. Linux vs. windows etc. You also gotta make sure that all your projects reference the same version of the dependencies. Again this is bull. If you have multiple teams this is a lot of coordination and overhead. We moved to SOA to get away from some of these.
I understand the data is heavier than the code, and it’s much more efficient to put the code next to the data itself, but is this the only way to achieve scale? Do you absolutely have to sacrifice Seperation of Concern when working with the amount of data Hadoop is supposed to address.
For example, yes you can embed CLR into Sql server, but in practice this is reserved only for the truly serious bottle neck that could not be resolved in any other way. aka – it’s a hack or an anti-pattern if you want to call it that. Do this too much, and your product becomes heavily coupled to Microsoft Sql Server. You can’t just swap it out for Oracle or what not as the biz need changes. Not good.
Also in all of computing history we have always bring the data to the code, not the other way around. Eg. you load the data from database to Orm, to service, to memory, to cache, and then to the instruction set. This stood for a reason namely SOC
My question is, whether map & reduce + no sql is one of those cases where you just have to put the code next to the data, as opposed to loading the data as needed to the instruction set (eg. a load balanced service somewhere in the cloud).
It depends on how much data you have.
At some point it becomes unpractical to move data across the network. If you need to do a map reduce on several TB of data then even with a gigabit network, just the data transfer time is significant.
If your data is in the cloud, then you also need to take into consideration the cost of transfering data from the cloud service.