On our site, we record mouse movements and clicks, which with limited traffic is fine but I can imagine would slow things down when there is significantly more people browsing the website. I figure that having a replica database for long-term storage is a good idea (to my limited knowledge, this is how computer systems handle memory anyway). This way, during the nighttime we’ll clear out the short-term database and update the long-term one.
Are there any disadvantages, pitfalls, or general notes about doing this? And in the case that this is not an efficient method, is there a better way of doing this?
You haven’t mentioned what server you use, but I would suggest using a document store, like http://www.mongodb.org/ or http://couchdb.apache.org/, you can find a nice list here: http://nosql-database.org/
I wouldn’t think you would need to break your data into short term / long term, if you do run into performance issues, I would suggest thinking about scaling horizontally.
Another option would be to use a hosted service like Amazon’s SimpleDb, you could have your data stored in something like mongo initially and then have a few worker processes that move your data into simpledb in the background (you wouldn’t want to add the overhead of the API call to your web app).