Question

I have a function in a service that basically pulls the data from a few tables and builds an index table.

The problem is that I have over 6000 records, so it just takes forever and sometimes reaches the memory limit.

Is there a design pattern or a method where I can achieve this either by a batch process or a web service? Ideally I want to be able to keep updating the user with the progress.

Any ideas or pointers would be really appreciated.

Update:

The tables I have are basically tables for use with products on an e-commerce site. I have the following tables:

• Product (basic product information)
• Product Description (descriptions and meta information)
• Product Price (price information on a product)
• Product Image (image information on a product)
• Product Features (any product features)
• Product Options (any product options)

I then have a Product Index table that pulls some of the information from all these tables for the purpose of making it much quicker and easier to search and filter results without the need of lots of joins.

I need some guidance on the best way to go about developing a method / function / process / service for rebuilding the index table.

Answer 1

I can’t tell you how to rebuild your index but I can show you a method I’ve been using for dealing with large result sets.

$em = $this->getContainer()->get('doctrine')->getEntityManager();
$repo = $em->getRepository('MyBundle:Item');
$qb = $repo->createQueryBuilder('i') // build query ...

// get iterator for query; this way only a single row is
// hydrated at a time.
$it = $qb->getQuery()->iterate();
foreach ($it as $row) {
    // $row[0] will be the fetched entity.
    $ent = $row[0];

    // do stuff with entity ...
    // ...

    // Once you're done with the entity detach it to
    // save memory and speed up the doctrine ORM.
    // make sure you DO NOT use the detached entity
    // in any way after this call (unless you re-merge() it).
    $em->detach($row[0]);
}