Question

For my program I am creating an inverted index using data from the Twitter feed, however, when parsing and putting them into the mongodb, some funny problems occur.

The usual type of entry should be like this:

{"ax"=>1, "easyjet"=>1, "from"=>2}

However, when parsing some of the tweets they end up like this in the db:

{""=>{""=>{""=>{""=>{""=>{"giants"=>{"dhem"=>1, "giants"=>1, "giantss"=>1}}}}

I have got these lines that split a tweet up and increments the values in the db:

def pull_hash_tags(tweet, lang)
    hash_tags = tweet.split.find_all { |word| /^#.+/.match word }
    t = tweet.gsub(/https?:\/\/[\S]+/,"") # removing urls
    t = t.gsub(/#\w+/,"") # removing hash tags
    t = t.gsub(/[^0-9a-z ]/i, '') # removing non-alphanumerics and keeping spaces
    t = t.gsub(/\r/," ")
    t = t.gsub(/\n/," ")
    hash_tags.each { |tag| add_to_hash(lang, tag, t) }
end

def add_to_hash(lang, tag, t)
    t.gsub(/\W+/, ' ').split.each { |word| @db.collection.update({"_id" => lang}, {"$inc" => {"#{tag}.#{word}" => 1}}, { :upsert => true }) }
end

I’m trying to get normal words (with only alphanumeric characters) and no double spaces, and no carriage returns etc.

Answer 1

You should add t.strip! as it seems the problem might be leading/trailing whitespace.