We have a SQL server 2008 and one of the tables, say table A has the following characteristics:
- Every day we get several heterogeneous feeds from other systems with numerical data.
- Feeds are staged elsewhere, converted to a format compliant with A’s schema.
- Inserted into A.
-
Schema looks like:
<BusinessDate> <TypeId> <InsertDate> <AxisX> <AxisY> <Value>
The table has a variable number of rows. Essentially we have to purge it at the weekends otherwise the size affects performance. So size ranges from 3m-15m rows during the week. Due to some new requirements we expect this number to be increased by 10m by the end of 2012. So we would be talking about 10m-25m rows.
Now in addition
- Data in A never change. The middle tier may use A’s data but it will be a read only operation. But typically the middle tier doesn’t even care about the contents. It typically (not always but 80% of cases) runs stored procs to generate reports and delivers the reports in other systems.
- Clients of these table would typically want to do do long sequential reads for one business date and type. i.e. “get me all type 1 values for today”
- Clients will want to join this table with 3-5 more tables and then deliver reports to other systems.
- The above assumptions are not necessarily valid for all tables with which A is joined. For example we usually join A with a table B and do a computation like B.value*A.value. B.value is a volatile column.
Question
- A’s characteristics do sound very much like what HBase and other column oriented schemas can offer.
- However some of the joins are with volatile data.
Would you recommend migrating A to an HBase schema?
And also, if we were to move A I would assume we would also have to migrate B and other dependent tables which (on the contrary with A) are being used by several other places from the middle tier. Wouldn’t this be complicating things a lot?
25 Million rows doesn’t sound big enough to justify using HBase, although the usage pattern fits. You need a name node, a job tracker, a master and then your region servers, so you’ll be needing a minimum of maybe 5 nodes to run HBase in any reasonable way. Your rows are so small I’m guessing it’s maybe 10gb of data, so storing this across 5 servers seems like overkill.
If you do go this route (perhaps you want to store more than a week’s data at once) there are ways to integrate HBase with relational DBs. Hive, for example, provides ODBC/JDBC connectivity and can query HBase. Oracle and Teradata both provide integration between their relational DB software and non-relational storage. I know Microsoft has recently announced that they are dropping Dryad in favor of integrating with Hadoop, but I am not certain how far along that process is wrt SQL Server. And if all you need is “get a list of IDs to use in my SQL query” you can of course write something yourself easily enough.
I think HBase is very exciting, and there may be things you haven’t mentioned which would drive you towards it (e.g. high availability). But my gut says you can probably scale out your relational db much more cheaply than switching to HBase.