I have this query below that I’ve rewritten a dozen different ways, yet I am unable to get it optimized and loaded in under 8 seconds. If I can get it to 2s, that would be good. 1s or less would be optimal.
This query retrieves a list of books that are currently available for sale or trade, and performs a bit of filtering. This query takes about 9-10 seconds.
SELECT
listing.for_sale,
listing.for_trade,
MIN(listing.price) AS from_price,
MAX(listing.price) AS to_price,
IF (NOW() > CONVERT_TZ(listing.date_sale_expiration, '+00:00', '-7:00'), 1, 0) AS expired,
COUNT(listing.book_id) AS 'listing_count',
book.id AS 'book_id',
book.title AS 'book_title',
book.date_released AS 'date_released',
book.publisher AS 'publisher',
book.photo_cover AS 'photo_cover',
publisher.name AS 'publisher_name',
COALESCE((SELECT COUNT(*) FROM listing l1 WHERE l1.book_id = book.id AND l1.status IN ('in_active_deal', 'complete')), 0) AS 'number_sold',
(SELECT 1 FROM listing l2 WHERE l2.status = 'available' AND l2.book_id = book.id AND l2.member_id = 1234 LIMIT 1) AS 'hasListing',
(SELECT 1 FROM wishlist w1 WHERE w1.book_id = book.id AND w1.member_id = 1234 LIMIT 1) AS 'hasWishlist'
FROM listing
INNER JOIN member ON
listing.member_id = member.id
AND member.transaction_limit <> 0
AND member.banned <> 1
AND member.date_last_login > DATE_SUB(CURDATE(), INTERVAL 120 DAY)
INNER JOIN book ON
listing.book_id = book.id
AND book.released = 1
INNER JOIN publisher ON
book.publisher_id = publisher.id
WHERE
listing.type = 'book'
AND listing.status = 'available'
AND (listing.for_trade = 1 OR (listing.for_sale = 1 AND NOW() < COALESCE(CONVERT_TZ(listing.date_sale_expiration, '+00:00', '-7:00'), 0)))
AND (
EXISTS (SELECT 1 FROM listing l3 LEFT JOIN book b ON l3.book_id = b.id WHERE l3.member_id = 1234 AND b.publisher_id = book.publisher_id AND l3.status = 'available' AND l3.type = 'book' AND (l3.for_trade = 1 OR (l3.for_sale = 1 AND NOW() < COALESCE(CONVERT_TZ(l3.date_sale_expiration, '+00:00', '-7:00'), 0))) LIMIT 1)
OR member.publisher_only <> 1
OR member.id = 1234
)
AND (
EXISTS (SELECT 1 FROM wishlist w WHERE w.member_id = member.id AND w.type = 'book' AND (w.type, w.book_id) IN (SELECT l4.type, l4.book_id FROM listing l4 WHERE 1234 = l4.member_id AND l4.status = 'available' AND (l4.for_trade = 1 OR (l4.for_sale = 1 AND NOW() < COALESCE(DATE_SUB(l4.date_sale_expiration, INTERVAL 7 HOUR), 0)))) LIMIT 1)
OR member.wishlist_only <> 1
OR member.id = 1234
)
GROUP BY
book.id
ORDER BY
book.date_released DESC
LIMIT 30;
These are my tables:
CREATE TABLE `listing` (
`id` int(10) unsigned NOT NULL auto_increment,
`member_id` int(10) unsigned NOT NULL,
`type` enum('book','audiobook','accessory') NOT NULL,
`book_id` int(10) unsigned default NULL,
`audiobook_id` int(10) unsigned default NULL,
`accessory_id` int(10) unsigned default NULL,
`date_created` datetime NOT NULL,
`date_modified` datetime NOT NULL,
`date_sale_expiration` datetime default NULL,
`status` enum('available','in_active_deal','complete','deleted') NOT NULL default 'available',
`for_sale` tinyint(1) unsigned NOT NULL default '0',
`for_trade` tinyint(1) unsigned NOT NULL default '0',
`price` decimal(10,2) default NULL,
`condition` tinyint(1) unsigned default NULL,
`notes` varchar(255) default NULL,
PRIMARY KEY (`id`),
KEY `ix_accessory` (`accessory_id`,`member_id`,`type`,`status`),
KEY `ix_book` (`book_id`,`member_id`,`type`,`status`),
KEY `ix_member` (`member_id`,`status`,`date_created`),
KEY `ix_audiobook` (`audiobook_id`,`member_id`,`type`,`status`),
KEY `ix_status` (`status`,`accessory_id`,`for_trade`,`member_id`)
) ENGINE=MyISAM AUTO_INCREMENT=281724 DEFAULT CHARSET=utf8
CREATE TABLE `member` (
`id` int(10) unsigned NOT NULL auto_increment,
`email` varchar(200) NOT NULL,
`screen_name` varchar(25) default NULL,
`date_last_login` datetime default NULL,
`wishlist_only` tinyint(1) unsigned NOT NULL default '1',
`platform_only` tinyint(1) unsigned NOT NULL default '0',
`transaction_limit` smallint(6) NOT NULL default '5',
`banned` tinyint(1) unsigned NOT NULL default '0',
`notes` text,
PRIMARY KEY (`id`),
KEY `ix_email` (`email`),
KEY `ix_screen_name` (`screen_name`),
KEY `ix_transaction_limit` (`transaction_limit`)
) ENGINE=MyISAM AUTO_INCREMENT=50842 DEFAULT CHARSET=utf8
CREATE TABLE `publisher` (
`id` int(10) unsigned NOT NULL auto_increment,
`name` varchar(128) NOT NULL,
`date_updated` datetime default NULL,
PRIMARY KEY (`id`),
KEY `ix_name` (`name`)
) ENGINE=MyISAM AUTO_INCREMENT=129 DEFAULT CHARSET=utf8
CREATE TABLE `book` (
`id` int(10) unsigned NOT NULL auto_increment,
`publisher_id` int(10) unsigned default NULL,
`name` varchar(128) NOT NULL,
`description` text,
`keywords` varchar(200) default NULL,
`date_released` varchar(10) default NULL,
`genre` varchar(50) default NULL,
`subgenre` varchar(50) default NULL,
`author` varchar(100) default NULL,
`date_updated` datetime default NULL,
`photo_cover` varchar(50) default NULL,
`weight_oz` decimal(7,2) default NULL,
`released` tinyint(2) NOT NULL default '0',
PRIMARY KEY (`id`),
KEY `ix_genre` (`genre`),
KEY `ix_name` (`name`),
KEY `ix_released` (`released`,`date_released`),
KEY `ix_keywords` (`keywords`)
) ENGINE=MyISAM AUTO_INCREMENT=87329 DEFAULT CHARSET=utf8
CREATE TABLE `wishlist` (
`id` int(10) unsigned NOT NULL auto_increment,
`member_id` int(10) unsigned NOT NULL,
`type` enum('book','audiobook','accessory') NOT NULL,
`book_id` int(10) unsigned default NULL,
`audiobook_id` int(10) unsigned default NULL,
`accessory_id` int(10) unsigned default NULL,
`date_created` datetime NOT NULL,
`date_modified` datetime NOT NULL,
PRIMARY KEY (`id`),
KEY `ix_accessory` (`accessory_id`,`member_id`,`type`),
KEY `ix_book` (`book_id`,`member_id`,`type`),
KEY `ix_member_accessory` (`member_id`,`accessory_id`),
KEY `ix_member_date_created` (`member_id`,`date_created`),
KEY `ix_member_book` (`member_id`,`book_id`),
KEY `ix_member_audiobook` (`member_id`,`audiobook_id`),
KEY `ix_audiobook` (`audiobook_id`,`member_id`,`type`)
) ENGINE=MyISAM AUTO_INCREMENT=241886 DEFAULT CHARSET=utf8
And here is the result when I run EXPLAIN:
+----+--------------------+-----------+----------------+---------------------------------------------------------------------------------------+----------------------+---------+------------------------------------+-------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+-----------+----------------+---------------------------------------------------------------------------------------+----------------------+---------+------------------------------------+-------+----------------------------------------------+
| 1 | PRIMARY | member | range | PRIMARY,ix_transaction_limit | ix_transaction_limit | 2 | NULL | 19617 | Using where; Using temporary; Using filesort |
| 1 | PRIMARY | listing | ref | ix_game,ix_member,ix_status | ix_member | 5 | live_database001.member.id,const | 7 | Using where |
| 1 | PRIMARY | book | eq_ref | PRIMARY,ix_released | PRIMARY | 4 | live_database001.listing.book_id | 1 | Using where |
| 1 | PRIMARY | publisher | eq_ref | PRIMARY | PRIMARY | 4 | live_database001.book.publisher_id | 1 | |
| 6 | DEPENDENT SUBQUERY | w | ref | ix_member_accessory,ix_member_date_created,ix_member_book,ix_member_publisher | ix_member_accessory | 4 | live_database001.member.id | 6 | Using where |
| 7 | DEPENDENT SUBQUERY | l4 | index_subquery | ix_book,ix_member,ix_status | ix_book | 11 | func,const,func,const | 1 | Using where |
| 5 | DEPENDENT SUBQUERY | l3 | ref | ix_book,ix_member,ix_status | ix_member | 5 | const,const | 63 | Using where |
| 5 | DEPENDENT SUBQUERY | b | eq_ref | PRIMARY | PRIMARY | 4 | live_database001.l3.book_id | 1 | Using where |
| 4 | DEPENDENT SUBQUERY | w1 | ref | ix_book,ix_member_accessory,ix_member_date_created,ix_member_game,ix_member_publisher | ix_book | 9 | func,const | 1 | Using where; Using index |
| 3 | DEPENDENT SUBQUERY | l2 | ref | ix_book,ix_member,ix_status | ix_book | 9 | func,const | 1 | Using where; Using index |
| 2 | DEPENDENT SUBQUERY | l1 | ref | ix_book,ix_status | ix_book | 5 | func | 10 | Using where; Using index |
+----+--------------------+-----------+----------------+--------------------------------------------------------------------------------------+----------------------+---------+------------------------------------+-------+----------------------------------------------+
This brings me to a couple questions:
1. The member table is using ix_transaction_limit, and as a result is searching through 19k+ rows. Since I am specifying a member.id, shouldn’t this be using PRIMARY and shouldn’t the rows be 1? How can I fix this?
2. How does the key_len affect the performance?
3. I’ve seen other complex queries which dealt with 100’s of millions of rows take less time. How is it that only 19k rows are taking so long?
(I’m still very green with MySQL Optimization, so I’d really love to understand the how’s & why’s)
Any suggestions small or big is greatly appreciated, thank you in advance!
After dropping the index on the member table, I was still having the same problems. In fact, that made it even worse. So my ultimate solution was to completely re-write the query from scratch.
And consequently, changing the order in the conditionals made a big difference as well. So moving the
and member_id = 1234and theand wishlish_only <> 1up above the subquery was a huge improvement.Thanks for all your help!