I need some inspiration for a solution…
We are running an online game with around 80.000 active users – we are hoping to expand this and are therefore setting a target of achieving up to 1-500.000 users.
The game includes a highscore for all the users, which is based on a large set of data. This data needs to be processed in code to calculate the values for each user.
After the values are calculated we need to rank the users, and write the data to a highscore table.
My problem is that in order to generate a highscore for 500.000 users we need to load data from the database in the order of 25-30.000.000 rows totalling around 1.5-2gb of raw data. Also, in order to rank the values we need to have the total set of values.
Also we need to generate the highscore as often as possible – preferably every 30 minutes.
Now we could just use brute force – load the 30 mio records every 30 minutes, calculate the values and rank them, and write them in to the database, but I’m worried about the strain this will cause on the database, the application server and the network – and if it’s even possible.
I’m thinking the solution to this might be to break up the problem some how, but I can’t see how. So I’m seeking for some inspiration on possible alternative solutions based on this information:
- We need a complete highscore of all ~500.000 teams – we can’t (won’t unless absolutely necessary) shard it.
- I’m assuming that there is no way to rank users without having a list of all users values.
- Calculating the value for each team has to be done in code – we can’t do it in SQL alone.
- Our current method loads each user’s data individually (3 calls to the database) to calculate the value – it takes around 20 minutes to load data and generate the highscore 25.000 users which is too slow if this should scale to 500.000.
- I’m assuming that hardware size will not an issue (within reasonable limits)
- We are already using memcached to store and retrieve cached data
Any suggestions, links to good articles about similar issues are welcome.
Interesting problem. In my experience, batch processes should only be used as a last resort. You are usually better off having your software calculate values as it inserts/updates the database with the new data. For your scenario, this would mean that it should run the score calculation code every time it inserts or updates any of the data that goes into calculating the team’s score. Store the calculated value in the DB with the team’s record. Put an index on the calculated value field. You can then ask the database to sort on that field and it will be relatively fast. Even with millions of records, it should be able to return the top n records in O(n) time or better. I don’t think you’ll even need a high scores table at all, since the query will be fast enough (unless you have some other need for the high scores table other than as a cache). This solution also gives you real-time results.