Question

After using a keyword API to get popular keywords and phrases, I get also a lot of “dirty” terms with too much extra words (“the”, “a”, etc.).

I’d also like to isolate names in search terms.

Is there a Ruby library to clean up keyword lists? Does such an algorithm exist at all?

Answer 1

You’re talking about “stopwords”, which are articles of speech, such as “the” and “a”, plus words that are encountered so often that they are worthless.

Stopword lists exist; Wordnet has one if I remember right and there might be one in Lingua or the Ruby Wordnet for Ruby or readablity modules, but really they’re pretty easy to generate yourself. And, you probably need to since the junk words vary depending on a particular subject matter.

The easiest thing to do is run a preliminary pass with several sample documents and split your text into words, then loop over them, and for each one increment a counter. When you’re finished look for the words that are two to four letters long and are disproportionately higher counts. Those are good candidates for stopwords.

Then run passes over your target documents, splitting the text like you did previously, counting occurrences as you go. You can either ignore words in your stopword list and not add them to your hash, or process everything then delete the stopwords.

text = <<EOT
You have reached this web page by typing "example.com", "example.net","example.org"
or "example.edu" into your web browser.

These domain names are reserved for use in documentation and are not available
for registration. See RFC 2606, Section 3.
EOT

# do this against several documents to build a stopword list. Tweak as necessary to fine-tune the words.
stopwords = text.downcase.split(/\W+/).inject(Hash.new(0)) { |h,w| h[w] += 1; h }.select{ |n,v| n.length < 5 }

print "Stopwords => ", stopwords.keys.sort.join(', '), "\n"

# >> Stopwords => 2606, 3, and, are, by, com, edu, for, have, in, into, net, not, or, org, page, rfc, see, this, use, web, you, your

Then, you’re ready to do some keyword gathering:

text = <<EOT
You have reached this web page by typing "example.com", "example.net","example.org"
or "example.edu" into your web browser.

These domain names are reserved for use in documentation and are not available
for registration. See RFC 2606, Section 3.
EOT

stopwords = %w[2606 3 and are by com edu for have in into net not or org page rfc see this use web you your]

keywords = text.downcase.split(/\W+/).inject(Hash.new(0)) { |h,w| h[w] += 1; h }
stopwords.each { |s| keywords.delete(s) }

# output in order of most often seen to least often seen.
keywords.keys.sort{ |a,b| keywords[b] <=> keywords[a] }.each { |k| puts "#{k} => #{keywords[k]}"}
# >> example => 4
# >> names => 1
# >> reached => 1
# >> browser => 1
# >> these => 1
# >> domain => 1
# >> typing => 1
# >> reserved => 1
# >> documentation => 1
# >> available => 1
# >> registration => 1
# >> section => 1

After you’ve narrowed down your list of words you can run the candidates through WordNet and find synonyms, homonyms, word relations, strip plurals, etc. If you’re doing this to a whole lot of text you’ll want to keep your stopwords in a database where you can continually fine-tune them. The same thing applies to your keywords, because from those you can start to determine tone and other semantic goodness.