Question

In my rails app, I have a “terms” model, that stores a term (a keyword), and the frequency with which it appears in a particular document set (an integer). Whenever a new document gets added to the set, I parse out the words, and then I need to either insert new terms, and their frequency, into the terms table, or I need to update the frequency of an existing term.

The easiest way to do this would be to do a find, then if it’s empty do an insert, or if it’s not empty, increment the frequency of the existing record by the correct amount. That’s two queries per word, however, and documents with high word counts will result in a ludicrously long list of queries. Is there a more efficient way to do this?

Answer 1

You can do this really efficiently, actually. Well, if you’re not afraid to tweak Rails’s default table layout a bit, and if you’re not afraid to generate your own raw SQL…

I’m going to assume you’re using MySQL for your database (I’m not sure what other DBs support this): you can use INSERT … ON DUPLICATE KEY UPDATE to do this.

You’ll have to tweak your count table to get it to work, though – “on duplicate key” only refers to the primary key, and Rails’s default ID, which is just an arbitrary number, won’t help you. You’ll need to change your primary key so that it identifies what makes each record unique – in your case, I’d say PRIMARY KEY(word, document_set_id). This might not be supported by Rails by default, but there’s at least one plugin, and probably a couple more, if you don’t like that one.

Once your database is set up, you can build one giant insert statement, and throw that at MySQL, letting the “on duplicate key” part of the query take care of the nasty existence-checking stuff for you (NOTE: there are plugins to do batch inserts, too, but I don;t know how they work – particularly in regards to “on duplicate key”):

counts = {}
#This is just demo code!  Untested, and it'll leave in punctuation...
@document.text.split(' ').each do |word|
    counts[word] ||= 0
    counts[word] += 1
end

values = []
counts.each_pair do |word, count|
    values << ActiveRecord::Base.send(:sanitize_sql_array, [
        '(?, ?, ?)',
        word,
        @document.set_id,
        count
    ])
end

#Massive line - sorry...
ActiveRecord::Base.connection.execute("INSERT INTO word_counts (word, document_set_id, occurences) VALUES ${values.join(', ')} ON DUPLICATE KEY UPDATE occurences = occurences + VALUES(occurences)")

And that’ll do it – one SQL query for the entire new document. Should be much faster, half because you’re only running a single query, and half because you’ve sidestepped ActiveRecord’s sluggish query building.

Hope that helps!