How does logging in a Hadoop job work? Using SLF4J and Logback, what sort of configuration would I need to see all the logging output in one place? Does STDOUT for a Hadoop job get collated by the JobTracker?
Share
Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
How does logging in a Hadoop job work? Using SLF4J and Logback, what sort of configuration would I need to see all the logging output in one place? Does STDOUT for a Hadoop job get collated by the JobTracker?
The log directory on each datanode contains a sub-directory
userlogs. This contains sub-diretories for recent map-task attempts. That is for each instance of a map task. Since the task attempt contains the job-id in its name you can find out what logs where created by a specific job.The task attempt directories contain the files:
These contain the respective outputs.
You can access task logs from the JobTracker Web-GUI by navigating from a listed Job to its tasks, clicking on a task and selecting its output.