Question

I have an application to transfer data from remote systems to HDFS using map reduce . I however am lost when I have to deal with isues like network failure .. That is , when a connection from remote data source is lost and data is no longer accessible to my mapreduce application. I can always restart the job but when data is huge then restarting is an expensive option . I know the mapreduce would create temp folder but will it put data there ? Can I read that data out and then Can I somehow start reading the rest of the data ?

Answer 1

A mapreduce job can write arbitrary files, not only the ones managed by Hadoop.

Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
out = fs.create(new Path(fileName));

using this code you create arbitrary files which work like normal files in the local filesystem. Then, you manage connection exceptions such that when a source is unaccessible you nicely close the file and record somewhere (e.g. in HDFS itself) that happened an interruption and at which point.
In the case of FTP, you could write just the list of file paths and folders. When a job finish to download a file, write its path on the downloaded list, and when an entire folder is downloaded write the folder path, so in case of resume you will not have to traverse a directory content to check that all files were downloaded.

At the program startup, on the other hand, it will check this file to decide whether the previous attempt failed and, in case, where to start the download.

In general, Hadoop will kill your program if it’s not writing/reading anything for a timeout. Your application can tell it to wait but in general is not good to have an idle job, so it’s better to end the job nicely instead that waiting for the network to work again.

You can also create your own filewriter, this way:

conf.setOutputFormat(MyOwnOutputFormat.class);

your filewriter could save its own temporary files in the format you prefer, so if the application crashes you know how files are saved.
HDFS saves files with chunks of 64MB by default, and when a job fails you may not even have a temporary file unless you use your own writer.

This is a generic solution, it depends on which is the source of data (ftp, samba, http…) and its support to download resumes.

EDIT: in case of FTP, you could just use csync to syncronize a FTP server with your local filesystem, and hdfs-fuse to mount a HDFS filesystem. It works when you have many small files.