Question

I’m a bit confused here… I’m being offered to get into a project, where would be an array of certain sensors, that would give off reading every millisecond ( yes, 1000 reading in a second ). Reading would be a 3 or 4 digit number, for example like 818 or 1529. This reading need to be stored in a database on a server and accessed remotely.

I never worked with such big amounts of data, what do you think, how much in terms of MBs reading from one sensor for a day would be?… 4(digits)x1000x60x60x24 … = 345600000 bits … right ? about 42 MB per day… doesn’t seem too bad, right?

therefor a DB of, say, 1 GB, would hold 23 days of info from 1 sensor, correct?

I understand that MySQL & PHP probably would not be able to handle it… what would you suggest, maybe some aps? azure? oracle?

Answer 1

3 or 4 digit number =

4 bytes if you store it as a string.
2 bytes storing it as a 16bit (0-65535) integer

1000/sec -> 60,000/minute -> 3,600,000/hour, 86,400,000/day

as string: 86,400,000 * 4 bytes = 329megabytes/day
as integer:86,400,000 * 2bytes = 165megabytes/day

Your DB may not perform too well under that kind of insert load, especially if you’re running frequent selects on the same data. optimizing a DB for largescale retrieval slows things down for fast/frequent inserts. On the other hand, inserting a simple integer is not exactly a “stressful” operation.

You’d probably be better off inserting into a temporary database, and do an hourly mass copy into the main ‘archive’ database. You do your analysis/mining on that main archive table, with the understanding that its data will be up to 1 hour stale.

But in the end, you’ll have to benchmark variations of all this and see what works best for your particular usage case. There’s no “you must do X to achieve Y” type advice in databaseland.