I’ve installed CDH4 on a ubuntu 12 LTS server successfully in the amazon cloud (1 server). I used Cloudera Manager free edition to install the software and had no errors).
I have a program that uses the java API to load a file from my home computer to HDFS in the cloud. I would like to know why this program fails and how to fix it.
Configuration conf = new Configuration();
conf.set("fs.defaultFS", "hdfs://node01:8020");
FileSystem fs = FileSystem.get(conf);
Path targetPath = new Path("/users/<username>/myfile.txt");
Path sourcePath = new Path("/home/<username>/myfile.txt");
fs.copyFromLocalFile(false,true,sourcePath,targetPath);
I get the following error (namenode log):
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/<username>/myfile.txt could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1322)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2170)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:471)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:297)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44080)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
Then i upload my code to the cloud and run the code locally (uploading a file from the local fs to hdfs) there aren’t any errors. It only happens when I run the code from my personal computer…
When i use the cli ‘hadoop fs -put’ command on my cloud server, I get no errors when writing to hdfs. I can also upload files using Hue. I’ve done some reading and found that this problem occurs when there isn’t enough disk space, but I have plenty for both dfs and non-dfs (see report below). I can successfully read the hadoop filesystem with the java API from my home computer, and I can even connect and read/write from HBase using the API. All ports are open to my IP on this server. File permissions have been checked. After the program fails, I see the file I tried to upload in hdfs , but the contents are blank (similar to this post: https://groups.google.com/a/cloudera.org/forum/?fromgroups=#!topic/cdh-user/XWA-3H0ekYY )
here is the output from hdfs dfsadmin -report
Configured Capacity: 95120474112 (88.59 GB)
Present Capacity: 95120474112 (88.59 GB)
DFS Remaining: 95039008768 (88.51 GB)
DFS Used: 81465344 (77.69 MB)
DFS Used%: 0.09%
Under replicated blocks: 177
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 1 (1 total, 0 dead)
Live datanodes:
Name: privateip:port (node01)
Hostname: node01
Rack: /default
Decommission Status : Normal
Configured Capacity: 95120474112 (88.59 GB)
DFS Used: 81465344 (77.69 MB)
Non DFS Used: 0 (0 KB)
DFS Remaining: 95039008768 (88.51 GB)
DFS Used%: 0.09%
DFS Remaining%: 99.91%
Last contact: Sun Jan 27 03:01:53 UTC 2013
I’ve resolved the problem-
I was connecting to hadoop from my home machine not on the hadoop local network. Apparently when you do this, the namenode tells my home machine to write to the datanode using the datanode’s private IP. Not being on the same network, my home machine coulnd’t connect to the datanode creating this error.
I resolved the problem by creating a VPN connection from my home network to the hadoop network and now everything work.