Question

So basically I have just benchmarked my update_feeds controller and found that the amount of sql queries that are run are shocking. I am looking for a way to optimise the process of fectching multiple feeds and then insert the data into a table (title and url)

Currently there are 193 Feeds in the DB that I fetch the URL’s for and then I process these one by one checking and insert data of them into another table. The problem is simplepie goes through and inserts every item.

So I am ending up with QUERIES: 3297
Total Execution Time 202.8051

I am looking for a way to optimise this process does anyone have any tips? I will post some code. Thanks.

Controller for fetching the feeds

          $this->output->set_profiler_sections($sections);
        $this->load->library('simplepie');
        //$this->simplepie->cache_location = BASEPATH .'cache';
        $this->load->model('FeedModel');
        $this->load->model('FeedItemModel');
        $feeds = $this->FeedModel->get_feed_update_urls();
        foreach ($feeds as $feed_id => $feed_url) {
            $this->simplepie->set_feed_url($feed_url);
            //$this->simplepie->set_cache_duration(0);
$this->simplepie->set_timeout(0);
            $this->simplepie->init();
            $items = $this->simplepie->get_items();
            foreach ($items as $item) {
                $this->FeedItemModel->load($feed_id, md5($item->get_id()));
                $this->FeedItemModel->link = $item->get_permalink();
                $this->FeedItemModel->title = $item->get_title();
                $this->FeedItemModel->created_time = $item->get_date('Y-m-d H:i:s');
                $this->FeedItemModel->save();
            }
        }

Model for inserting feeds

function save() {
        if ($this->_id !== false) {
            $this->db->query('UPDATE feed_items SET link=?, title=?, created_time=? WHERE id=?', array($this->link, $this->title, $this->created_time, $this->_id));
        } else {
            $this->db->query('INSERT INTO feed_items(feed_id, remote_id, link, title, created_time, updated_time) VALUES (?, ?, ?, ?, ?, NOW()) ON DUPLICATE KEY UPDATE remote_id=remote_id', array($this->feed_id, $this->remote_id, $this->link, $this->title, $this->created_time, $this->remote_id));
            $this->_id = $this->db->insert_id();
        }
    }

Answer 1

Instead of loading every single time and then saving, you should instead do an update (and then insert when update doesn’t find rows). That will reduce by a single query per feed item.