Question

i’ve stored procedure, that check against MySql server if can insert or update records.
for start time it’s so fast, after 5-10 minutes it’s so slowly..
i run for 3000 records and the execute of the sp so bad, after 1-1.5 hours it’s finished…
my ask, how to can i to improve that??

thanks.

The stored procedure:

DELIMITER $$ 
DROP PROCEDURE IF EXISTS test.SPInsertUpdateCity$$ 
CREATE DEFINER=root@localhost PROCEDURE SPInsertUpdateCity( 
in SP_CityName VARCHAR(100) charset utf8, 
in SP_CitySynonyms mediumtext charset utf8, 
in SP_CityNumberPostOffice varchar(100), 
in SP_CityUpdatedDate date) 
BEGIN 
    if(not exists(select CityID from city where CityName = SP_CityName)) then 
        insert into city(CityName, CitySynonyms, CityNumberPostOffice ,UpdatedDate) 
        values(SP_CityName, CONCAT(',',SP_CitySynonyms, ','),SP_CityNumberPostOffice,SP_CityUpdatedDate);
    else if((exists(select cityId from city 
    where 
        CityName = SP_CityName and 
        (UpdatedDate < SP_CityUpdatedDate or UpdatedDate = SP_CityUpdatedDate))) and 
        not exists(SELECT CitySynonyms FROM city 
        WHERE 
            CitySynonyms in(select CitySynonyms from city where CitySynonyms like CONCAT('%,',SP_CitySynonyms,',%')))) 
            then 
            update city set 
                CitySynonyms = CONCAT(CitySynonyms,SP_CitySynonyms,','),
                UpdatedDate = SP_CityUpdatedDate; 
    end if; 
    end if; 
END$$ 
DELIMITER ; 

Answer 1

I normally work with MSSQL but from what I can tell its your elseif section that giving you the hit. You are hitting the table 3 times everytime its run when it could be twice.

This is your original query reformatted for easy comparison.

    if(not exists(  select  CityID 
                    from    city 
                    where   CityName = SP_CityName)) then
        insert into city(   CityName, CitySynonyms, CityNumberPostOffice ,UpdatedDate) 
                values  (   SP_CityName, CONCAT(',',SP_CitySynonyms, ','), SP_CityNumberPostOffice, SP_CityUpdatedDate);
    else if((exists(    select  cityId 
                        from    city      
                        where   CityName = SP_CityName and 
                                (UpdatedDate < SP_CityUpdatedDate or UpdatedDate = SP_CityUpdatedDate))) and          
            not exists( SELECT CitySynonyms 
                        FROM city          
                        WHERE CitySynonyms in ( select CitySynonyms 
                                                from city 
                                                where CitySynonyms like CONCAT('%,',SP_CitySynonyms,',%')))) then
        update city set CitySynonyms = CONCAT(CitySynonyms,SP_CitySynonyms,','),                 
                        UpdatedDate = SP_CityUpdatedDate;      
    end if;      
    end if;  

This is the new one

if(not exists(  select  CityID 
                from    city 
                where   CityName = SP_CityName)) then
    insert into city(   CityName, CitySynonyms, CityNumberPostOffice ,UpdatedDate) 
            values  (   SP_CityName, CONCAT(',',SP_CitySynonyms, ','), SP_CityNumberPostOffice, SP_CityUpdatedDate);
else if((exists(    select  cityId 
                    from    city      
                    where   CityName = SP_CityName and 
                            UpdatedDate <= SP_CityUpdatedDate)) and          
        not exists( select  CitySynonyms 
                    from    city 
                    where   CitySynonyms like CONCAT('%,',SP_CitySynonyms,',%'))) then
    update city set CitySynonyms = CONCAT(CitySynonyms,SP_CitySynonyms,','),                 
                    UpdatedDate = SP_CityUpdatedDate;      
end if;      
end if;  

There are a few other things that could be done so speed up the query further.
A like clause is much slower than an equals. If possible, change CitySynonyms like CONCAT(‘%,’,SP_CitySynonyms,’,%’) to CitySynonyms = SP_CitySynonyms.
Say you supply a Synonym starting with “D” the database can index seek straight to the D’s and will not even look at the other records.
Of course for that to work you will need an index on the table that includes the CitySynonyms column. If you use a Like clause, the database will run a instring() check
against the column for every row in the table which is a major performance hit when dealing with a quite a few rows.

Think of it in terms of a phone directory, if I want a person with last name Smith, I would look up the index at the back of the book and find that the S’s start on page 600 (called seeking). I dont care about the first 599 pages and wont even look at any names on them. Without the index I would have to go over each page until I got to names starting with S then SM and so on (called scanning). But a LIKE clause is like searching for a last name that contains SMITH which could be any of the following (name from from Wiki) BlackSmith Coppersmith GoldSmith HammerSmith Smither Smithers ect. As you can see the names are all over the place. Every name will need to be checked to see if it contains SMITH. Its one of the most uncool things you can ask someone to do but we ask the database to stuff like that all the time and then ask why is it slow. 🙂

The other thing is the (UpdatedDate < SP_CityUpdatedDate or UpdatedDate = SP_CityUpdatedDate) which I have changed to UpdatedDate <= SP_CityUpdatedDate.
Is that check really needed? Because if you dont, you can remove that whole if exists section because you know it exist if execution reaches that point as the very first if check which returned false because a record exists. So now the else if is a single hit on the table instead of three.

if(not exists(  Select  CityID 
                from    city 
                where   CityName = SP_CityName)) then          

    insert into city(   CityName, CitySynonyms, CityNumberPostOffice ,UpdatedDate) 
            values  (   SP_CityName, CONCAT(',',SP_CitySynonyms, ','), SP_CityNumberPostOffice, SP_CityUpdatedDate);     

else if(not exists( select  CitySynonyms 
                    from    city 
                    where   CitySynonyms like CONCAT('%,',SP_CitySynonyms,',%'))) then              

    update city set CitySynonyms = CONCAT(CitySynonyms,SP_CitySynonyms,','),                 
                    UpdatedDate = SP_CityUpdatedDate;      
end if;      
end if;

EDIT – Added in response to questions in comments below.

Before I go any further, should

update city set CitySynonyms = CONCAT(CitySynonyms,SP_CitySynonyms,','), 
                UpdatedDate = SP_CityUpdatedDate;

be

update city set CitySynonyms = CONCAT(CitySynonyms,SP_CitySynonyms,','), 
                UpdatedDate = SP_CityUpdatedDate
where CityName = SP_CityName; 

Because as it is currently, it adding a synonym to every row in the database, when I think it should be updating only the one you are interested in.

As for the like statement. I think I have just realized whats going on. Correct me if I’m wrong. The CitySynonyms column is a large text field in your table, with multiple values in it separated by commas, and you are trying to find a match in that string using the like statement.

If so, Is it possible to change your table structure slightly? Which means moving the CitySynonyms column into a second table.

CREATE TABLE city ( CityID int(20) NOT NULL AUTO_INCREMENT, 
                    CityName varchar(100) NOT NULL, 
                    CityNumberPostOffice varchar(100) DEFAULT NULL, 
                    UpdatedDate date DEFAULT NULL, 
PRIMARY KEY (CityID) ) ENGINE=InnoDB DEFAULT CHARSET=utf8

CREATE TABLE CitySynonym ( CitySynonymID int(20) NOT NULL AUTO_INCREMENT,
                           CityID int(20) NOT NULL ,                    
                           CitySynonym varchar(100) NOT NULL,
PRIMARY KEY (CitySynonymID) ) ENGINE=InnoDB DEFAULT CHARSET=utf8

For each city, you can have multiple synonyms. It becomes much easier to use and maintain, not to mention that the LIKE clause will no longer be needed.

Example:
City Table
CityID    CityName
1         Brisbane
2         Syndney

City Synonym Table
CitySynoymnID    CityID  Synonym
1                1       BNE
2                1       BRISSY
3                2       SYD