Backdrop:
I have a file hierarchy of cvs files for multiple locations named by dates they cover …by month specifically. Each cvs file in the folder is named after the location.
eg’,
folder name: 2010-feb
contains:
location1.csv
location2.csv
Each CSV file holds records like this:
2010-06-28, 20:30:00 , 0
2010-06-29, 08:30:00 , 0
2010-06-29, 09:30:00 , 0
2010-06-29, 10:30:00 , 0
2010-06-29, 11:30:00 , 0
meaning of record columns ( column names ):
Date, time, # of sessions
I have a perl script that pulls the data from this mess and originally I was going to store it as json files, but am thinking a database might be more appropriate long term …comparing year to year trends …fun stuff like that.
Pt 2 – My question/problem:
So I now have a REST service that coughs up json with a test database. My question is [ I suck at db design ], how best to design a database backend for this?
I am thinking the following tables would suffice and keep it simple:
Location: (PK)location_code, name
session: (PK)id, (FK)location_code, month, hour, num_sessions
I need to be able to average sessions (plus min and max) for each hour across days of week in addition to days of week in a given month or months. I’ve been using perl hashes to do this and am trying to decide how best to implement this with a database.
Do you think stored procedures should be used?
As to the database, depending on info gathered here, it will be postgresql or sqlite.
If there is no compelling reason for postgresql I’ll stick with sqlite.
How and where should I compare the data to hours of operation. I am storing the hours
of operation in a yaml file. I currently ‘match’ the hour in the data to a hash from the yaml to do this. Would a database open simpler methods? I am thinking I would do this comparison as I do now then insert the data. Can be recalled with:
SELECT hour, num_sessions FROM session WHERE location_code=LOC1
Since only hours of operation are present, I do not need to worry about it.
Should I calculate all results as I do now then store as a stats table for
different ‘reports’? This, rather than processing on demand? How would this look?
Anyway …I ramble.
Thanks for reading!
Bubnoff
From what I read of SQLite, it offers the functions you need for doing the analysis you want (sum, avg, etc), and it looks like you’ll be doing that at your own api level as opposed to allowing an end user to do it themselves through an interface. So for the simple design + small dataset you have I would look at getting all your data into SQLite. I’d also put it in the format SQLite can understand natively, so that you can use its SQL functions without needing to convert anything first or without needing to create special functions to be used within SQL to do the conversion.
Aside from that, your design looks fine to me except for the month and hour fields. I would leave those as full date and time fields, or possibly combine them to just one date_time field if there’s an appropriate SQLite data type for that, and put the full date/time data in them (in case you’ll need it later). Then use SQLite time functions to extract the month and hour as appropriate from your full date/time fields. As a convenience, and if SQLite supports it, you could create calculated fields for month and hour in the session table, which would let you immediately return the data you’re looking for from a query, instead of explicitly calling time extraction functions in every query you want a month or hour for.
Also, don’t forget to put indexes on fields that you set criteria on in queries. You may not notice a difference with small data sets, but as your db gets bigger, they could make a huge difference.