Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
I am looking for ways to send works for multiple computers on my University computer grid.
Currently it is running Condor and also offers Hadoop.
My question is thus, should I try and interface with R to Hadoop or to the Conder for my projects?
For the discussion, let’s assume we are talking about Embarrassingly parallel tasks.
p.s: I’ve seen the resources described in the CRAN task views.
You can do both.
You can use HDFS for your data sets and Condor for your job scheduling. Using Condor to place executors on machines and HDFS + Hadoops Map-Reduce features to process your data (assuming your problem is map-reduce mappable). Then you’re using the most appropriate tool for the job: Condor is a job scheduler, and as such does that work better than Hadoop. And Hadoop’s HDFS and M-R framework are things Condor doesn’t have (but are really helpful for jobs running on Condor to use).
I would personally look at has HDFS to share data among jobs that run discretely as Condor jobs. Especially in a university environment, where shared compute resources are not 100% reliable and can come and go at will, Condor’s resilience in this type of set up is going to make getting work done a whole lot easier.