Question

i have to parse files

• logs (5gb)
• error logs (1gb)
• import data file (csv) (400mb)
• backup data (10gb)

etc…

my first thoughts is to parse these files in php then either remove what i don’t need and save them to my database. for the import file it’s running ok since it’S small

but when i want to check the backup data, the php fail and stop (i get different error, like memory error)

i would like to know whats the best way to parse and import these big files, preferably, i would like to have 1 script that manage all of them. is my php script can handle it? or i should do something else?

Answer 1

fread is pretty useful
http://php.net/manual/en/function.fread.php

it will parse line by line so that only one line or at max X bytes are in memory at a time.