Question

I’m trying to save the parsed data into MongoDB. I want to store every referrer URL. At present it is taking a lot of time to insert into MongoDB.

  class ReferrerDocument
    include MongoMapper::Document
    key :website_url, String
    key :referrer_url, String
    key :full_referrer_url, String
    key :count_of_appearance, Integer, :default => 1
    key :current_time, Time
    timestamps!

    def self.save_referrer_info(website_url, referrer_url, full_referrer_url, current_time)
      referrer_document = ReferrerDocument.last(:website_url => website_url,
                                                :referrer_url => referrer_url,
                                                :order => :id.asc)
      if(referrer_document.nil?)
        ReferrerDocument.create(:website_url => website_url,
                                :referrer_url => referrer_url,
                                :full_referrer_url => full_referrer_url,
                                :last_seen_at => 0,
                                :current_time => current_time)
      else
        count = referrer_document.count_of_appearance += 1
        last_seen_at = referrer_document.current_time.utc
        ReferrerDocument.create(:website_url => website_url,
                                :referrer_url => referrer_url,
                                :full_referrer_url => full_referrer_url,
                                :last_seen_at => last_seen_at,
                                :current_time => current_time,
                                :count_of_appearance => count)
      end
    end
end

Answer 1

Did you try to actually measure how long each part takes to execute?

From what I see, the only place that could possibly be slow is ReferrerDocument.last invocation.

How many documents you have in this collection? Do you have supporting indexes?
What mongodb query does ReferrerDocument.last translate to (you can see in development.log. I myself am a mongoid user)?
What does explain plan for this query look like?

To make this query most efficient you’ll need an index on {website_url: 1, referrer_url: 1, _id: 1}