Question

I have a process that imports a lot of data (950k rows) using inserts that insert 500 rows at a time. The process generally takes about 12 hours, which isn’t too bad. Normally doing a query on the table is pretty quick (under 1 second) as I’ve put (what I think to be) the proper indexes in place. The problem I’m having is trying to run a query when the import process is running. It is making the query take almost 2 minutes! What can I do to make these two things not compete for resources (or whatever)? I’ve looked into “insert delayed” but not sure I want to change the table to MyISAM.

Thanks for any help!

Answer 1

So I finally found the slowdown while searching during the import of my data. I had one query like this:

SELECT * FROM `properties` WHERE (state like 'Florida%') and (county like 'Hillsborough%') ORDER BY created_at desc LIMIT 0, 50

and when I ran an EXPLAIN on it, I found out it was scanning around 215,000 rows (even with proper indexes on state and county in place). I then ran an EXPLAIN on the following query:

SELECT * FROM `properties` WHERE (state = 'Florida') and (county = 'Hillsborough') ORDER BY created_at desc LIMIT 0, 50

and saw that it only had to scan 500 rows. Considering that the actual result set was something like 350, I think I identified the slowdown.

I’ve made the switch to not using “like” in my queries and am very happy with the snappier results.

Thanks to everyone for your help and suggestions. They are much appreciated!