I’m stuck inbetween a rock and a hard place with an application I was making. Basically the program generates a very large dataset(its over 10 gigs and exceeds memory on the server and its hard to transfer this dataset from a dictionary to other servers later on). I decided to use a database to store the data but am running into a problem in which as I spawn more proceses they do not use more than 10% of its available core but the mysql service is always around 60-100%.
The core that is using mysql is running between 60-100%. I have taken basic mysql courses but am no way an expert on it and I’m really just using it to store/distribute local variables of the program. Any suggestions of how I can troubleshoot or even if this problem is possible to solve? One idea I had was although my memory cannot store the entire database I plan to write, I would love for the server to use some memory to process some requests and then maybe write them as memory gets full(right now mysql is using less than 1% of server memory).
As you know my goal, I have no allegiance to any specific type of system so if mysql is not a good solution, I am happy to try another tool(I was going to use hbase, but went down the mysql road first). If anything else is better please let me know.
It all depends on what you are actually trying to do but you’ll need to provide a lot more information in order to get good advice.
What do you mean by large number of writes? 10? 100? 1000? per second?
What is the schema you are trying to write to? What indexes are implemented on the schema?
Do you have heavy reads to writes or is this all writes?
What are you doing during your write? Is it a single row you are writing or is it a number of operations wrapped in a transaction?
What is the server configuration? Memory? Disk? Version of mySQL.
Are you using Innodb for the database?
Now one thing you can try is to run SHOW FULL PROCESSLIST;. This will show you if there is some type of locking occurring – if you see a lot of the processes in a locked state, then look for one of the processes that is not locked and see if that may be causing locking. See http://dev.mysql.com/doc/refman/5.0/en/show-processlist.html for more info on this command.