I am beginner for HBase. I am currently not sure what influences HBase performance. First, I ran Hbase in a stand alone mode. I used a single machine to run a Mapreduce program to process 2 million text lines and output the results to a HBase table stored in local file system. it took about 1h40min. Then I changed to the pseudo distributed mode. Htable file is then stored in HDFS.All other things including program remain the same. Then it took more than 3 hours!! I am totally confused. Can anyone tell me why this happaned?
Another question, since I create 5 virtual machines as the virtual cluster on a Linux machine which is not very powerful(8G ram. 3GHz 4 core CPU). When I ran Mapreduce program month ago, I saw no big difference when I ran it on a single master or with 5 more slaves. So I assumed that the CPU might be the bottleneck but still I am not sure. Can anyone give answers for sure since I am not much familiar with Operating System? I am not sure whether it’s a IO bottleneck or CPU bottleneck. And I just run the program on the 6 machines cluster for inserting records into HBase table on a fully distributed mode. It was too slow so I killed the job.
For the first part of your question:
It is obvious that pseudo-distributed mode will run slower than standalone mode. In standalone, HBase uses the local filesystem for storage. In pseudo-distributed mode, HBase uses HDFS (which in turn uses the local filesystem), so you have an additional software layer for IO. Moreover, in pseudo-distributed mode, you might have multiple server threads in the same machine, performing things like replication (every data is copied a couple of times), which doesn’t happen in standalone.
In general, the more centralized things are (like in standalone), the better performance you will have, for small (enough) data. But distributed modes are important for large scale, since anyway one day you will hit the scalability limit of a single machine. Then in distributed modes we need to deal with things such as replication that eventually reduce overall performance.
About the second part of your question, you shouldn’t expect to have good performance in these settings. You have to consider that there are many software layers: HBase, HDFS, Java, virtual machine. Each of these layers adds overhead. With distributed slaves this gets even slower because all servers share the same hardware.
If you want to see some performance increase, consider using standalone mode in a Linux OS, not as a virtual machine. And then use a real distributed mode (not in virtual machines) in a real cluster (if you can, but even a couple of networked consumer computers will do). You should expect about 1K~3K Puts per second.