Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
I am wondering how PigStorage in Pig stores data to S3? Does it save output to HDFS and then copy them over? Or saving each reducer output to local directory of each reducer and then copying them over to S3? I guess this can’t be streaming since S3 supports only putting files or a directory?
My understanding is that each reducer writes its output locally and then copies the output to S3.
As you have correctly stated – since S3 doesn’t support streaming, the reducer can only copy its output once it has finished processing.