I am getting some issues in HBase. When I try to start hadoop , it starts without any issue.but HBase not works.DataNode , HRegionServer , secondaryNameNode etc all are working but HMaster is not running, when I start hbase.Sometimes it throws an exception “RegionServer is running as process #xyz .stop it first.Another strange thing is HBase logs are printing “Matser started” but when we check , it is not start.I am searching out the issue , but unable to understand. Guide me if you get a clue.Here the logs of HBase given below
Wed Apr 4 17:48:46 PKT 2012 Starting master on hbase
ulimit -n 1024
2012-04-04 17:48:48,804 INFO org.apache.hadoop.hbase.ipc.HBaseRpcMetrics: Initializing RPC Metrics with hostName=HMaster, port=60000
2012-04-04 17:48:49,185 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server Responder: starting
2012-04-04 17:48:49,191 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server listener on 60000: starting
2012-04-04 17:48:49,199 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60000: starting
2012-04-04 17:48:49,203 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 60000: starting
2012-04-04 17:48:49,204 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60000: starting
2012-04-04 17:48:49,213 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60000: starting
2012-04-04 17:48:49,213 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60000: starting
2012-04-04 17:48:49,271 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60000: starting
2012-04-04 17:48:49,272 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60000: starting
2012-04-04 17:48:49,275 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 60000: starting
2012-04-04 17:48:49,278 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 60000: starting
2012-04-04 17:48:49,280 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 6 on 60000: starting
2012-04-04 17:48:49,396 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.2-1031432, built on 11/05/2010 05:32 GMT
2012-04-04 17:48:49,400 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=hbase.ifkaar.com
2012-04-04 17:48:49,400 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.6.0_20
2012-04-04 17:48:49,400 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
2012-04-04 17:48:49,400 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
2012-04-04 17:48:49,400 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/opt/ifkaar/hbase-0.90.4/bin/../conf:/usr/lib/jvm/java-6-openjdk/lib/tools.jar:/opt/ifkaar/hbase-0.90.4/bin/..:/opt/ifkaar/hbase-0.90.4/bin/../hbase-0.90.4.jar:/opt/ifkaar/hbase-0.90.4/bin/../hbase-0.90.4-tests.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/activation-1.1.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/asm-3.1.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/avro-1.3.3.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/commons-cli-1.2.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/commons-codec-1.4.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/commons-configuration-1.6.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/commons-el-1.0.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/commons-httpclient-3.1.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/commons-lang-2.5.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/commons-logging-1.1.1.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/commons-net-1.4.1.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/core-3.1.1.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/guava-r06.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/hadoop-core-0.20.205.0.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/hadoop-gpl-compression-0.2.0-dev.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/jackson-core-asl-1.5.5.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/jackson-jaxrs-1.5.5.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/jackson-mapper-asl-1.4.2.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/jackson-xc-1.5.5.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/jasper-compiler-5.5.23.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/jasper-runtime-5.5.23.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/jaxb-api-2.1.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/jaxb-impl-2.1.12.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/jersey-core-1.4.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/jersey-json-1.4.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/jersey-server-1.4.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/jettison-1.1.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/jetty-6.1.26.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/jetty-util-6.1.26.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/jruby-complete-1.6.0.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/jsp-2.1-6.1.14.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/jsp-api-2.1-6.1.14.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/jsr311-api-1.1.1.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/log4j-1.2.16.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/protobuf-java-2.3.0.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/servlet-api-2.5-6.1.14.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/slf4j-api-1.5.8.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/slf4j-log4j12-1.5.8.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/stax-api-1.0.1.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/thrift-0.2.0.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/xmlenc-0.52.jar:/opt/ifkaar/hbase-0.90.4/bin/../lib/zookeeper-3.3.2.jar
2012-04-04 17:48:49,401 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/i386/client:/usr/lib/jvm/java-6-openjdk/jre/lib/i386:/usr/lib/jvm/java-6-openjdk/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/jni:/lib:/usr/lib
2012-04-04 17:48:49,401 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2012-04-04 17:48:49,401 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2012-04-04 17:48:49,401 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
2012-04-04 17:48:49,401 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=i386
2012-04-04 17:48:49,401 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=2.6.32-38-generic
2012-04-04 17:48:49,401 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=ifkaar
2012-04-04 17:48:49,401 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/home/ifkaar
2012-04-04 17:48:49,401 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/opt/ifkaar/hbase-0.90.4/bin
2012-04-04 17:48:49,403 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=180000 watcher=master:60000
2012-04-04 17:48:49,469 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181
2012-04-04 17:48:49,506 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
2012-04-04 17:48:49,688 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1367d66702b0000, negotiated timeout = 180000
2012-04-04 17:48:49,740 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=Master, sessionId=hbase.ifkaar.com:60000
2012-04-04 17:48:49,784 INFO org.apache.hadoop.hbase.metrics: MetricsString added: revision
2012-04-04 17:48:49,784 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsUser
2012-04-04 17:48:49,784 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsDate
2012-04-04 17:48:49,784 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsUrl
2012-04-04 17:48:49,784 INFO org.apache.hadoop.hbase.metrics: MetricsString added: date
2012-04-04 17:48:49,784 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsRevision
2012-04-04 17:48:49,800 INFO org.apache.hadoop.hbase.metrics: MetricsString added: user
2012-04-04 17:48:49,800 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsVersion
2012-04-04 17:48:49,800 INFO org.apache.hadoop.hbase.metrics: MetricsString added: url
2012-04-04 17:48:49,800 INFO org.apache.hadoop.hbase.metrics: MetricsString added: version
2012-04-04 17:48:49,801 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
2012-04-04 17:48:49,802 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
2012-04-04 17:48:49,802 INFO org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
2012-04-04 17:48:49,939 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Master=hbase.ifkaar.com:60000
2012-04-04 17:48:52,204 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 0 time(s).
2012-04-04 17:48:53,205 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 1 time(s).
2012-04-04 17:48:54,206 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 2 time(s).
2012-04-04 17:48:55,207 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 3 time(s).
2012-04-04 17:48:56,207 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 4 time(s).
2012-04-04 17:48:57,208 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 5 time(s).
2012-04-04 17:48:58,209 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 6 time(s).
2012-04-04 17:48:59,210 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 7 time(s).
2012-04-04 17:49:00,211 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 8 time(s).
2012-04-04 17:49:01,212 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:54310. Already tried 9 time(s).
2012-04-04 17:49:01,216 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.
java.net.ConnectException: Call to localhost/127.0.0.1:54310 failed on connection exception: java.net.ConnectException: Connection refused
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1095)
at org.apache.hadoop.ipc.Client.call(Client.java:1071)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at $Proxy6.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:118)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:222)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:187)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1328)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:65)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1346)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:244)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
at org.apache.hadoop.hbase.util.FSUtils.getRootDir(FSUtils.java:364)
at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:81)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:346)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:282)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:604)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:434)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:560)
at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1202)
at org.apache.hadoop.ipc.Client.call(Client.java:1046)
... 17 more
2012-04-04 17:49:01,221 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
2012-04-04 17:49:01,221 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping service threads
2012-04-04 17:49:01,222 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60000
2012-04-04 17:49:01,222 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60000: exiting
2012-04-04 17:49:01,222 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 60000: exiting
2012-04-04 17:49:01,222 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60000: exiting
2012-04-04 17:49:01,223 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60000: exiting
2012-04-04 17:49:01,223 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60000: exiting
2012-04-04 17:49:01,223 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60000: exiting
2012-04-04 17:49:01,223 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 6 on 60000: exiting
2012-04-04 17:49:01,223 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 60000: exiting
2012-04-04 17:49:01,224 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 60000: exiting
2012-04-04 17:49:01,224 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60000: exiting
2012-04-04 17:49:01,224 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 60000
2012-04-04 17:49:01,228 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder
2012-04-04 17:49:01,300 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2012-04-04 17:49:01,301 INFO org.apache.zookeeper.ZooKeeper: Session: 0x1367d66702b0000 closed
2012-04-04 17:49:01,301 INFO org.apache.hadoop.hbase.master.HMaster: HMaster main thread exiting
Wed Apr 4 17:49:38 PKT 2012 Stopping hbase (via master)
I think, this line from the log,
says that it is tying to connect to namenode at port 54310 and is failing.
Can you please verify by typing this in your browser and say me the accurate port you are using?
You’ll see something like NameNode ‘localhost:?’ as the first line and the ‘?‘ is your namenode port.