Home/ Questions/Q 7984947
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-04T11:25:56+00:00

We are designing a table for ad-hoc analysis that will capture umpteen value fields

  • 0

We are designing a table for ad-hoc analysis that will capture umpteen value fields over time for claims received. The table structure is essentially (pseudo-ish-code):

   table_huge (
     claim_key int not null,
     valuation_date_key int not null,
     value_1 some_number_type,
     value_2 some_number_type,
     [etc...],
     constraint pk_huge primary key (claim_key, valuation_date_key)
   );

All value fields all numeric. The requirements are: The table shall capture a minimum of 12 recent years (hopefully more) of incepted claims. Each claim shall have a valuation date for each month-end occurring between claim inception and the current date. Typical claim inception volumes range from 50k-100k per year.

Adding all this up I project a table with a row count on the order of 100 million, and could grow to as much as 500 million over years depending on the business’s needs. The table will be rebuilt each month. Consumers will select only. Other than a monthly refresh, no updates, inserts or deletes will occur.

I am coming at this from the business (consumer) side, but I have an interest in mitigating the IT cost while preserving the analytical value of this table. We are not overwhelmingly concerned about quick returns from the Table, but will occasionally need to throw a couple dozen queries at it and get all results in a day or three.

For argument’s sake, let’s assume the technology stack is, I dunno, in the 80th percentile of modern hardware.

The questions I have are:

  • Is there a point at which the cost-to-benefit of indices becomes excessive, considering a low frequency of queries against high-volume tables?
  • Does the SO community have experience with +100M row tables and can
    offer tips on how to manage?
  • Do I leave the database technology problem to IT to solve or should I
    seriously consider curbing the business requirements (and why?)?

I know these are somewhat soft questions, and I hope readers appreciate this is not a proposition I can test before building.

Please let me know if any clarifications are needed. Thanks for reading!

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    First of all: Expect this to “just work” if leaving the tech problem to IT – especially if your budget allows for an “80% current” hardware level.

    I do have experience with 200M+ rows in MySQL on entry-level and outdated hardware, and I was allways positivly suprised.

    Some Hints:

    • On monthly refresh, load the table without non-primary indices, then create them. Search for the sweet point, how many index creations in parallell work best. In a project with much less date (ca. 10M) this reduced load time compared to the naive “create table, then load data” approach by 70%

    • Try to get a grip on the number and complexity of concurrent queries: This has influence on your hardware decisions (less concurrency=less IO, more CPU)

    • Assuming you have 20 numeric fields of 64 bits each, times 200M rows: If I can calculate correctly, ths is a payload of 32GB. Trade cheap disks against 64G RAM and never ever have an IO bottleneck.

    • Make sure, you set the tablespace to read only

    • 0