I have a website scraping script that is storing data into few MySQL tables. The records in the main table are referred from the other ones. The script would need around 5 hours to be executed, but it always stops working after around 70 minutes (set_time_limit() is set ofc). I tried to run it from both browser and cron, the result was the same.
Since the script stops at a random time, it can lead to database inconsistency. The main table record can be stored, but not the subordinant tables records using mysql_insert_id() from the main one.
Is there any better way than to delete the last record from the main table and all the referring records to the row from the other tables when running the script?
You could wrap all your queries for a given record in a transaction, and commit it at the end. That way, if the script dies partway through a set of queries, the changes will be rolled back and the data will be kept consistent.