What tools/methods do you recommend to diagnose and profile MySQL in live production server?
My goal to test alternative ways of scaling up the system and see their influence on read/write timing, memory, CPU load, disk access etc. and to find bottlenecks.
First of all you should set up some kind of monitoring with e.g.:
Other may helpful tools: mytop innotop mtop maatkit
In addtion you should enable logging slow-queries in your my.cnf.
Befor you start to tune/change parameters you should create some kind of test plan and compare the before/after results to see wether your changes made sense or not.