Question

I am importing approx 350,000 lines from a CSV file into MySQL using PHP (Symfony Framework if that matters) I am reading in each line and inserting into the SQL table as I go. This will be a daily process – and the number of lines will increase.

I have to translate 2 of the columns to a number (which is a foreign key). I want to know the most efficient way of doing it. I am experimenting with using a lookup table in MySQL and also an array in PHP – but I am wondering if there is a more efficient way.

There are approx 60 different combinations the 2 columns could be, here is an example of the translation table I have

Feature         Type          CODE
ANZIRL          Voice Call    8
BT2DT2          Data Call     6
BT2IL2          Voice Call    1
BT2UK2          Voice Call    2
DG2DG2          Voice Call    4
DG2DT2          Data Call     3
DG2EC           Voice Call    1
DG2EDQ          Voice Call    5
DG2EZY          Voice Call    7

And the CSV will be as follows

DG2DG2,x,x,x,Voice Call,x,x
DG2DT2,x,x,x,Data Call,x,x

etc…

I need to insert that into my table as

x,4,x,x
x,3,x,x

etc ….

Answer 1

I would batch import the CSV file into MySQL as is (i.e. create a table that mirrors CSV field layout), and then run a query against the CSV import table to get the foreign key code.

Something like:

INSERT into sometable (code,other,fields)
SELECT code, other, fields
FROM codes c
JOIN csvtable ct
WHERE ct.feature = c.feature AND ct.type = c.type

Basic idea is to not perform processing on row-by-row basis in PHP or MySQL, but to get all the data into MySQL and then run some version of above query to efficiently do the foreign key translation in 1-step.

If you go row-by-row with 350,000+ records, that will be a fairly slow process, php is not terribly efficient…