Question

I’m trying to create or open a file to store some output in HDFS, but I’m getting a NullPointerException when I call the exists method in the second to last line of the code snippet below:

DistributedFileSystem dfs = new DistributedFileSystem();
Path path = new Path("/user/hadoop-user/bar.txt");
if (!dfs.exists(path)) dfs.createNewFile(path);
FSDataOutputStream dos = dfs.create(path);

Here is the stack trace:

java.lang.NullPointerException
        at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:390)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:667)
        at ClickViewSessions$ClickViewSessionsMapper.map(ClickViewSessions.java:80)
        at ClickViewSessions$ClickViewSessionsMapper.map(ClickViewSessions.java:65)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227)
        at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2209)

What could the problem be?

Answer 1

I think the preferred way of doing this is:

Configuration conf = new Configuration();
conf.set("fs.default.name", "hdfs://mynamenodehost:9000");
FileSystem fs = FileSystem.get(conf);
Path path = ...

That way you don’t tie your code to a particular implementation of FileSystem; plus you don’t have to worry about how each implementation of FileSystem is initialized.