Question

I am working on a script that routinely uploads a CSV file into a MySQL database. The issue is that one of the fields in the CSV file contains user generated text, which may include quotations and other characters which are unfriendly to MySQL.

I have determined that the most efficient way to upload the CSV file is via MySQL’s ‘LOAD DATA INFILE’ command. Here is the command as it appears in MyPHPAdmin when I upload the CSV:

LOAD DATA LOCAL INFILE '/home/myfolder/tmp/property_re_1.csv' REPLACE INTO TABLE `markers`
    FIELDS TERMINATED BY ';' ENCLOSED BY '"' ESCAPED BY '\\'
    LINES TERMINATED BY '\r\n' # 2 rows affected.

Simply loading the CSV with this command does not work, as the process terminates as soon as it runs into a ‘”‘ and the “ESCAPED BY ‘\'” doesn’t appear to be serving its purpose.

In this case, I think I may have to escape the quotations in the CSV file programmatically with PHP first. Then load the “Escaped” file into MySQL with the LOAD DATA INFILE command in SQL.

I am sure this is a common problem that has a “best practice” solution. Essentially, my script needs to “clean up” a CSV file before loading it into a MYSQL table.

Here is a link to the actual SQL table and CSV file I am working with to help wiser minds wrap their heads around this one: https://www.dropbox.com/sh/4iq10i51qlqyq8q/UjEQwvXKDA

Thanks for your help in advance.

Answer 1

The data file on your dropbox has fields delimited by ,, optionally enclosed by ", lines delimited by \n and containing a header row; whereas your statement above is for a file with fields delimited by ;, always enclosed by ", lines delimited by \r\n and not containing any header row.

That said, it’s also for a file named data.csv whereas the one on Dropbox is named property_re_1.csv.

mysql> CREATE TABLE IF NOT EXISTS `markers` (
    ->   `L_ListingID` int(20) NOT NULL,
    ->   `L_Class` int(5) NOT NULL,
    ->   `L_Type_` int(10) NOT NULL,
    ->   `L_Status` varchar(10) NOT NULL,
    ->   `L_AskingPrice` float NOT NULL,
    ->   `L_Remarks` text NOT NULL,
    ->   `L_Address` varchar(50) NOT NULL,
    ->   `L_City` text NOT NULL,
    ->   `L_State` varchar(10) NOT NULL,
    ->   `LO1_OrganizationName` varchar(65) NOT NULL,
    ->   `LA1_AgentLicenseID` int(20) NOT NULL,
    ->   `LA1_UserFirstName` varchar(50) NOT NULL,
    ->   `LA1_UserLastName` varchar(50) NOT NULL,
    ->   `L_PictureCount` int(4) NOT NULL,
    ->   PRIMARY KEY (`L_ListingID`)
    -> ) ENGINE=MyISAM DEFAULT CHARSET=latin1;
Query OK, 0 rows affected (0.21 sec)

mysql> LOAD DATA LOCAL INFILE 'property_re_1.csv' 
    -> REPLACE INTO TABLE `markers`
    -> FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\\'
    -> LINES TERMINATED BY '\n' IGNORE 1 LINES;
Query OK, 315 rows affected (0.01 sec)
Records: 315  Deleted: 0  Skipped: 0  Warnings: 0