Home/ Questions/Q 6689341
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-26T05:30:29+00:00

I am working with an application which has a 3 tables each with more

  • 0

I am working with an application which has a 3 tables each with more than 10mm records and larger than 2GB.

Every time data is inserted there’s at least one record added to each of the three tables and possibly more.
After every INSERT a script is launched which queries all these tables in order to extract data relevent to the last INSERT (let’s call this the aggregation script).

What is the best way to divide the DB in smaller units and across different servers so that the load for each server is manageable?

Notes:
1. There are in excess of 10 inserts per second and hence the aggregation script is run the same number of times.
2. The aggregation script is resource intensive
3. The aggregation script has to be run on all the data in order to find which one is relevant to the last insert
4. I have not found a way of somehow dividing the DB into smaller units
5. I know very little about distributed DBs, so please use very basic terminology and provide links for further reading if possible

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    There are two answers to this from a database point of view.

    1. Find a way of breaking up the database into smaller units. This is very dependent on the use of your database. This is really your best bet because it’s the only way to get the database to look at less stuff at once. This is called sharding:
      http://en.wikipedia.org/wiki/Shard_(database_architecture)

    2. Have multiple “slave” databases in read only mode. These are basically copies of your database (with a little lag). For any read only queries where that lag is acceptable, they access these databases across the code in your entire site. This will take some load off of the master database you are querying. But, it will still be resource intensive on any particular query.

    From a programming perspective, you already have nearly all your information (aside from ids). You could try to find some way of using that information for all your needs rather than having to requery the database after insert. You could have some process that only creates ids that you query first. Imagine you have tables A, B, C. You would have other tables that only have primary keys that are A_ids, B_ids, C_ids. Step one, get new ids from the id tables. Step two, insert into A, B, C and do whatever else you want to do at the same time.

    Also, general efficiency/performance of all queries should be reviewed. Make sure you have indexes on anything you are querying. Do explain on all queries you are running to make sure they are using indexes.

    This is really a midlevel/senior dba type of thing to do. Ask around your company and have them lend you a hand and teach you.

    • 0