I have a table, namely X, with about 64 columns that stores online financial transactions.
Every day millions of records are inserted into X. About 16 columns of X are queryable, that is to say, many system reports need to filter X’s data based on the values of these 16 columns.
Having DB indexes for all of these 16 columns makes the insert action too slow. On the other hand, having no index on some columns makes some reports too slow as well.
So, here is the question. How I design the table X and its indexes to get the best performance on insert and report? I use oracle 11g DBMS.
Try to partition the table. Instead of creating one gigantic table, create one per day, week or month and a view which joins all of them for queries. Alternatively, Oracle has support for partitioned tables (but whether that’s available depends on your version of Oracle).
When you insert data, insert into the correct partition. That way, the index to update will be much smaller. The drawback is that you will need a lot more space for indexes because the index values will be duplicated.
On the positive side, queries might be much faster since the DB can read tables in parallel when they are on different disks.
Also note that SQL databases don’t scale to just any size. Consider a clustered or cloud database instead. They have other drawbacks but they can handle any number of records (as long as you have enough physical space for the servers, that is).