Question

I am using DoctrineExtensions for managing a nested tree(nested tree behavior). I have to import 10-20K records from a csv file and store them in the database.

I created a service for this task which makes a bunch of new categories(tree nodes), sets the parents of these categories and persists them and in the end I call EntityManager#flush method to save the created categories into the database.

According to my logic, the process should go like this:

1. The extension calculates the data needed for each node(right, left, level etc.)

2. Doctrine makes a query in which it inserts all the data, similarly to this:

   INSERT INTO table (field1, field2, parent_id, root, left, ...) VALUES
   (fields of 1st record),
   (fields of 2st record),
   ...
   

However, Doctrine inserts each record separately, additionally root, left and right values get updated during the flush causing a huge amount of update queries to be needed.

For inserting 70 categories, I needed ~250 queries, when I wanted to add 500 categories, the program terminated due to memory shortage.

How could I optimize the process? Of course I could write it from scratch, but I’d really love to use this superb extension.

Answer 1

My solution was to use materialized path tree instead of nested tree. I know, it’s not a real solution for this problem, however it works efficiently. If you have a better solution, please post it, and I might accept it as the solution.