Question

We created a import script which imports about 120GB of data into a MySQL database. The data is saved in a few hunderd directories (all are seperated databases). Each directory contains files with table structures and table data.

The issue being; it works on my local machine with a subset of the actual data, but when the import is ran on the server (Which takes a few days). Not all the tables are created (even tables that are tested locally). The odd thing is that the script, when ran on the server does not show any errors on the creation of the tables.

Here is on a high level how the script works:

1. Find all directories that represent a database
2. Create all databases
3. Per database loop through the tables: create table, fill table

Added the code on gist: https://gist.github.com/3349872

Answer 1

Auch, the problem was in the code. Amazingly stupid mistake.

When testing the code on a subset of all the files all table information and table content where available. When a table count not be created the function enters a logging statement and than returns. On the real data this was a the mistake because there are files with no data and no structure so after creating a few tables this creation of the tables of a certain database went wrong and did a return and so didn’t create the other tables.