My Use Case:
I want to import the large data from EC2 through SQOOP into the Hive. Imported data in Hive will get processed in Hive by applying some algorithm and will generate some result (in table form, in Hive only). And generated result will be exported back to Ec2 again through SQOOP only.
I am new to Amazon Web Services and want to implement this use case with the help of AWS EMR. I have implemented it on local machine.
I have read some links related to AWS EMR for launching the instance and about what is EMR, How it works and etc…
I have some doubts about EMR like:
1) EMR uses S3 Buckets, which holds Input and Output data Hadoop Processing (in the form of Objects). —> I didn’t get How to store the data in the form of Objects on S3 (My data will be files)
2) As already said I have implemented a task for my use case in Java. So If I create the JAR of my program and create the Job Flow with Custom JAR. Will it be possible to implement like this or do need to do some thing extra for that?
3) As I said in my Use Case that I want to export my result back to Ec2 with the help of SQOOP. Does EMR have support of SQOOP?
–edited part
4) Also I will import my data from SQL Server daily/weekly as my data in SQL Server get updated daily/weekly. If I think to import that data on S3 and give it to Hive then How can I do that? (Because Hive stores its data on HDFS under /user/hive/warehouse directory).
How can I make link to S3 and /user/hive/warehouse directory in HDFS.
Please reply me with your answer as soon as possible. I want to do this as early as possible.
many Thanks.
It is possible to install Sqoop on AWS EMR. You are not required to use S3 to store files and can use the local (temporary) HDFS instead. After you have Sqoop installed, you can import your data with it into HDFS, run your calculations in HDFS, then export your data back out using Sqoop again.
Here’s an article I wrote about how to install Sqoop on AWS EMR:
http://blog.kylemulka.com/2012/04/how-to-install-sqoop-on-amazon-elastic-map-reduce-emr/