Home/ Questions/Q 8397725
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-09T20:52:11+00:00

I would love to hear some opinions or thoughts on a mysql database design.

  • 0

I would love to hear some opinions or thoughts on a mysql database design.

Basically, I have a tomcat server which recieves different types of data from about 1000 systems out in the field. Each of these systems are unique, and will be reporting unique data.

The data sent can be categorized as frequent, and unfrequent data. The unfrequent data is only sent about once a day and doesn’t change much – it is basically just configuration based data.

Frequent data, is sent every 2-3 minutes while the system is turned on. And represents the current state of the system.

This data needs to be databased for each system, and be accessible at any given time from a php page. Essentially for any system in the field, a PHP page needs to be able to access all the data on that client system and display it. In other words, the database needs to show the state of the system.

The information itself is all text-based, and there is a lot of it. The config data (that doesn’t change much) is key-value pairs and there is currently about 100 of them.

My idea for the design was to have 100+ columns, and 1 row for each system to hold the config data. But I am worried about having that many columns, mainly because it isn’t too future proof if I need to add columns in the future. I am also worried about insert speed if I do it that way. This might blow out to a 2000row x 200column table that gets accessed about 100 times a second so I need to cater for this in my initial design.

I am also wondering, if there is any design philosophies out there that cater for frequently changing, and seldomly changing data based on the engine. This would make sense as I want to keep INSERT/UPDATE time low, and I don’t care too much about the SELECT time from php.

I would also love to know how to split up data. I.e. if frequently changing data can be categorised in a few different ways should I have a bunch of tables, representing the data and join them on selects? I am worried about this because I will probably have to make a report to show common properties between all systems (i.e. show all systems with a certain condition).

I hope I have provided enough information here for someone to point me in the right direction, any help on the matter would be great. Or if someone has done something similar and can offer advise I would be very appreciative. Thanks heaps 🙂

~ Dan

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    I’ve posted some questions in a comment. It’s hard to give you advice about your rapidly changing data without knowing more about what you’re trying to do.

    For your configuration data, don’t use a 100-column table. Wide tables are notoriously hard to handle in production. Instead, use a four-column table containing these columns:

    SYSTEM_ID  VARCHAR    System identifier
POSTTIME   DATETIME   The time the information was posted
NAME       VARCHAR    The name of the parameter
VALUE      VARCHAR    The value of the parameter

    The first three of these columns are your composite primary key.

    This design has the advantage that it grows (or shrinks) as you add to (or subtract from) your configuration parameter set. It also allows for the storing of historical data. That means new data points can be INSERTed rather than UPDATEd, which is faster. You can run a daily or weekly job to delete history you’re no longer interested in keeping.

    (Edit if you really don’t need history, get rid of the POSTTIME column and use MySQL’s nice extension feature INSERT ON DUPLICATE KEY UPDATE when you post stuff. See http://dev.mysql.com/doc/refman/5.0/en/insert-on-duplicate.html)

    If your rapidly changing data is similar in form (name/value pairs) to your configuration data, you can use a similar schema to store it.

    You may want to create a “current data” table using the MEMORY access method for this stuff. MEMORY tables are very fast to read and write because the data is all in RAM in your MySQL server. The downside is that a MySQL crash and restart will give you an empty table, with the previous contents lost. (MySQL servers crash very infrequently, but when they do they lose MEMORY table contents.)

    You can run an occasional job (every few minutes or hours) to copy the contents of your MEMORY table to an on-disk table if you need to save history.

    (Edit: You might consider adding memcached http://memcached.org/ to your web application system in the future to handle a high read rate, rather than constructing a database design for version 1 that handles a high read rate. That way you can see which parts of your overall app design have trouble scaling. I wish somebody had convinced me to do this in the past, rather than overdesigning for early versions. )

    • 0