Question

Ok, I have a query that I just can’t seem to get optimized. Maybe I am doing too much in MySQL and should delegate more to PHP. The query as it is takes about a minute, but I really need it to be much faster than that. Below is a hypothetical structure that should give you an idea of what I am trying to accomplish.

You will notice that I am supplying the query with a PHP variable, $bookList, which contains a list of book_id’s owned by the current user in a comma delimited format.

For this example, assume that the user has 70 book_id’s assigned to his library ($bookList) and he wants to find out which stores share the most books in common with his library. Each store has 70 books, but there are 200,000+ stores. Stores with less than 50% in common will be filtered out.

Table 1: my_books

id: mediumint
book_id: smallint

Table 2: store_books

store_id: mediumint
book_id: smallint
index: store_id
index: book_id

Table 3; stores

id: mediumint
name: varchar(50)
primary: id

Query:

SELECT count(s.book_id) AS commonBooks, s.id
                 FROM store_books AS sb
                 INNER JOIN stores AS s ON s.id = sb.id
                 WHERE sb.book_id IN ($bookList) 
                 GROUP BY sb.store_id 
                 HAVING commonBooks > 35
                 ORDER BY commonBooks

Thanks in advance!

Answer 1

Borrowing from @Joe Stefanelli’s answer, use the temp table.
Make sure you have the primary index on bl.book_id

BTW: You seem to have an error in your query:

SELECT count(*) AS commonBooks, s.id
FROM store_books AS sb
INNER JOIN stores AS s ON s.id = sb.store_id  -- <<sb.store_id, not sb.id
WHERE sb.book_id IN (SELECT bl.book_id FROM tempBookList)
GROUP BY sb.store_id 
HAVING commonBooks > 35
ORDER BY commonBooks DESC

It’s a little known fact that MySQL can only use one index per (sub)select.
Make sure you have a composite index in table store_book
Also count(*) is (sometimes) faster than count(afield) (and never slower)

Table 2: store_books

store_id: mediumint
book_id: smallint
index: (store_id, book_id)  <<-- composite primary index.
index: (book_id)

This query will run faster in InnoDB than MyISAM because InnoDB can use covering indexes to resolve this query and will not need to read the data in the actual tables.