Question

I’m fetching aroung 6k articles from the Magento database. Traversing through them in beginning is very fast (0 seconds, just some ms) and gets slower and slower. The loop takes about 8 hours to run and in the end each loop in the foreach takes about 16-20 seconds ! It seems like mysql is getting slower and slower in the end, but I cannot explain why.

$product = Mage::getModel('catalog/product');
$data = $product->getCollection()->addAttributeToSelect('*')->addAttributeToFilter('type_id', 'simple'); 
$num_products = $product->getCollection()->count();
echo 'exporting '.$num_products."\n";

print "starting export\n";
$start_time = time();
foreach ($data as $tProduct) {
    // doing some stuff, no sql !
}

Does anyone know why it is so slow ? Would it be faster, just to fetch the ids and selecting each product one by one ?

The memory usage of the script running this code has a constant memory usage of:

  VIRT   RES   SHR S CPU% MEM%
  680M  504M  8832 R 90.0  6.3

Regards, Alex

Answer 1

Oh, well, Shot in the dark time. If you are running Magento 1.4.x.x, previous to 1.4.2.0, you have a memory leak that displays exactly this symptom as it eats up more and more memory, leading eventually to memory exhaustion. Profile exports that took 3-8 minutes under 1.3.x.x will now take 2-5 hours if it doesn’t throw an error before completion. Another symptom is exports that fail without finishing and without giving any indication of why the window freezes or gives some sort of funky completion message with no output.

The Array Of Death(tm) has been noted and here’s the official repair in the new version. Maybe Data Will Flow again!

Excerpt from 1.4.2.0rc1 /lib/Varien/Db/Select.php that has been patched for memory leak

public function __construct(Zend_Db_Adapter_Abstract $adapter)
{
   parent::__construct($adapter);
   if (!in_array(self::STRAIGHT_JOIN_ON, self::$_joinTypes)) {
      self::$_joinTypes[] = self::STRAIGHT_JOIN_ON;
      self::$_partsInit = array(self::STRAIGHT_JOIN => false) + self::$_partsInit;
   }
}

Excerpt from 1.4.1.1 /lib/Varien/Db/Select.php with memory leak

public function __construct(Zend_Db_Adapter_Abstract $adapter)
{
   parent::__construct($adapter);
   self::$_joinTypes[] = self::STRAIGHT_JOIN_ON;
   self::$_partsInit = array(self::STRAIGHT_JOIN => false) + self::$_partsInit;
}