I have 2 main tables in My-SQL database.
Table1 is -> Master
Table2 is -> Stock
Master table has around 500K (half million) rows, each row is unique than the other. Each column of master table has a foreign key (sample attached below)
Stock table has around 20K rows at the moment; it can go up to 300K in near future.
Problem
My problem is that I am performing SQL query on these tables, which takes unknown time at the moment. So would like to know, how I could improve the performance of MySQL database to speed the execution time of the SQL query.
SQL Query does following:
Search Master table -> Search Stock table -> Update Master table
I am executing the SQL query via PHP file, that I run in browser to complete these above steps. Because the Master table has more than 500K records I call only 1,000 records at a time, and do the above steps, again iterate to next batch of 1,000 records and do this till the end of Master table.
I am achieving this by using —
for loop for to Search table; it makes around 5390 cycle in total
while loop to Search Stock and Update Master table.
Testing
For testing purpose I reduced the Stock table with just 100 records and kept Master with all 539K records to check the execution time.
It takes around 140 ~ 150 seconds.
Script
I have attached my script below
<?php
set_time_limit(36000);
$dbhost='localhost';
$dbuser='root';
$dbpass='';
$conn = mysql_connect($dbhost, $dbuser, $dbpass);
mysql_select_db("pmaster",$conn);
if (!$conn) { die('Could not connect : '.mysql_error()); }
$trecssql = "select count(distinct `Id`) Total from `Master`";
$trecs = mysql_fetch_assoc(mysql_query($trecssql));
$trecs = $trecs['Total'];
$recspt = 1000; // Records Per Transactions 539000
$trecs = ceil($trecs/$recspt);
$startrow = 1;
$endrow = 1000;
//$trecs = 50; // Comment it to work as normal.
$start_time = microtime(true);
Here is the loop over the batches
for ($i=1; $i<=$trecs; $i++) {
Here is the query that fetches the batch of master rows
$mastersql = "select `Id`,
`Attribute1`, `Attribute2`,
`Attribute3`, `Attribute4`, `Attribute5`
from `Master`
where Id between ".$startrow." and ".$endrow;
$this_mastersql = mysql_query($mastersql);
$updatesql = '';
Here is the loop that processes the batch
while($master_rec = mysql_fetch_assoc($this_mastersql)) {
Here is a query that summarizes the contents of the Stock for each item in the Master.
$searchsql = "select min(Price) minprice,
avg(Price) avgprice,
max(Price) maxprice,
Currency from `Stock`
where Name1 = '".$master_rec['Attribute1']."'
AND Name2 = '".$master_rec['Attribute2']."'
AND Name3 = '".$master_rec['Attribute3']."'
AND Name4 = '".$master_rec['Attribute4']."'
AND Name5 = '".$master_rec['Attribute5']."';";
$this_searchsql = mysql_query($searchsql);
$search_rec = mysql_fetch_assoc($this_searchsql);
Here is the query that updates the Master table.
$updatesql .= "update `Master`
set `MinP` = '".$search_rec['minprice']."',
`AvgP` = '".$search_rec['avgprice']."',
`MaxP` = '".$search_rec['maxprice']."',
`Currency` = '".$search_rec['currency']."'
where Id = '".$master_rec['Id']."';";
}
mysql_query($updatesql);
$startrow = $endrow+1;
$endrow = $startrow+999;
}
$end_time = microtime(true);
echo 'Updated <br /><br />';
echo "Scripts Execution time <br />";
echo "Time in Hours : Minutes : Seconds <br />"
.gmdate("H:i:s", $time_elapsed = $end_time - $start_time);
echo "<br /> <br /> Time in Seconds ".$time_elapsed;
?>
Tables Structure
Attached the sample of Master table in Excel file.
https://docs.google.com/open?id=0B8Oew7S4GzgiQk5tbUZKdXVEUms
CREATE TABLE IF NOT EXISTS `Attribute1` (
`Id` int(3) NOT NULL,
`Name` varchar(50) NOT NULL,
PRIMARY KEY (`Id`),
UNIQUE KEY `Name` (`Name`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
CREATE TABLE IF NOT EXISTS `Attribute2` (
`Id` int(10) NOT NULL,
`Name` decimal(3,2) NOT NULL,
PRIMARY KEY (`Id`),
UNIQUE KEY `Name` (`Name`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
CREATE TABLE IF NOT EXISTS `Attribute3` (
`Id` int(3) NOT NULL,
`Name` varchar(5) NOT NULL,
PRIMARY KEY (`Id`),
UNIQUE KEY `Name` (`Name`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
CREATE TABLE IF NOT EXISTS `Attribute4` (
`Id` int(3) NOT NULL,
`Name` varchar(20) NOT NULL,
PRIMARY KEY (`Id`),
UNIQUE KEY `Name` (`Name`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
CREATE TABLE IF NOT EXISTS `Attribute5` (
`Id` int(3) NOT NULL,
`Name` varchar(100) NOT NULL,
PRIMARY KEY (`Id`),
UNIQUE KEY `Name` (`Name`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
CREATE TABLE IF NOT EXISTS `Master` (
`Id` int(20) NOT NULL AUTO_INCREMENT,
`Attribute1` varchar(50) DEFAULT NULL,
`Attribute2` decimal(3,2) DEFAULT NULL,
`Attribute3` varchar(5) DEFAULT NULL,
`Attribute4` varchar(20) DEFAULT NULL,
`Attribute5` varchar(100) DEFAULT NULL,
`MinP` decimal(10,2) DEFAULT NULL,
`AvgP` decimal(10,2) DEFAULT NULL,
`MaxP` decimal(10,2) DEFAULT NULL,
`Currency` varchar(5) DEFAULT NULL,
PRIMARY KEY (`Id`),
KEY `Attribute1` (`Attribute1`),
KEY `Attribute2` (`Attribute2`),
KEY `Attribute3` (`Attribute3`),
KEY `Attribute4` (`Attribute4`),
KEY `Attribute5` (`Attribute5`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=1 ;
--
-- Constraints for table `Master`
--
ALTER TABLE `Master`
ADD CONSTRAINT `Master_ibfk_1` FOREIGN KEY (`Attribute1`) REFERENCES
`Attribute1` (`Name`) ON DELETE CASCADE ON UPDATE CASCADE,
ADD CONSTRAINT `Master_ibfk_2` FOREIGN KEY (`Attribute2`) REFERENCES
`Attribute2` (`Name`) ON DELETE CASCADE ON UPDATE CASCADE,
ADD CONSTRAINT `Master_ibfk_3` FOREIGN KEY (`Attribute3`) REFERENCES
`Attribute3` (`Name`) ON DELETE CASCADE ON UPDATE CASCADE,
ADD CONSTRAINT `Master_ibfk_4` FOREIGN KEY (`Attribute4`) REFERENCES
`Attribute4` (`Name`) ON DELETE CASCADE ON UPDATE CASCADE,
ADD CONSTRAINT `Master_ibfk_5` FOREIGN KEY (`Attribute5`) REFERENCES
`Attribute5` (`Name`) ON DELETE CASCADE ON UPDATE CASCADE;
CREATE TABLE `pmaster`.`Stock` (
`Id` int( 10 ) NOT NULL AUTO_INCREMENT ,
`Name1` varchar( 10 ) NOT NULL ,
`Name2` decimal( 5, 2 ) NOT NULL ,
`Name3` varchar( 5 ) NOT NULL ,
`Name4` varchar( 5 ) NOT NULL ,
`Name5` varchar( 40 ) NOT NULL ,
`OtherFields1` varchar( 20 ) NOT NULL ,
`OtherFields2` varchar( 25 ) NOT NULL ,
`SoOn` varchar( 15 ) NOT NULL ,
`Price` decimal( 15, 2 ) NOT NULL ,
`Currency` varchar( 5 ) NOT NULL ,
PRIMARY KEY ( `id` ) ,
) ENGINE = InnoDB DEFAULT CHARSET = latin1;
Currently the it takes ~ 140 to 150 seconds for just 100 records in Stock table, and as the records increased the time it takes manifolds, and with 20K records it takes ages, haven’t tried how much time it takes in total, but last time the script was running for more than 3 hours, I had to quit, because not a single record in Master table was updated.
Meaning it might be just at 1% to 2% of the completion, or may be less.
Any ideas friends how could this done in faster way.
If you’ll require the data for these database let me know would generate and upload here.
You should consider getting rid of your batches of 1000 records. You should consider using this query, just once, instead of your two queries. This query will yield a row for each UPDATE you need to do in your Master table.
This query yields one row for each group of rows in the Stock table which match the attributes in the Master table.
If you put a compound index on the Name items in the Stock table and another on the Attribute items in the Master table, this query should be reasonably efficient.