Question

I am trying to write a rake task to parse a file containing lots of twitter updates in json format.

The json file looks like this:

{ "_id" : { "$oid" : "50f82aeab4aa879861000000" }, "text" : "Golden Globes, lots of fashion messes...but glad Argo won. Really good movie, along w Moonrise Kingdom and Salmon Fishing Yeman.", "created_at" : { "$date" : 1358137508000 }, "id" : 290675893024747523, "user" : { "screen_name" : "Dpharmakis23", "id" : 852045842 } }
{ "_id" : { "$oid" : "50f82aeab4aa879861000001" }, "text" : "RT @RPLife: 2 Red Carpet HQs added and maybe my favorites of the night http://t.co/Fiq08m7i", "created_at" : { "$date" : 1358137507000 }, "id" : 290675892668211203, "user" : { "screen_name" : "dri_violet", "id" : 88925839 } }
{ "_id" : { "$oid" : "50f82aeab4aa879861000002" }, "text" : "After watching the Golden Globes it made me want to watch Les Mis again. It's gonna be a late night tonight.", "created_at" : { "$date" : 1358137507000 }, "id" : 290675892416573440, "user" : { "screen_name" : "JessBess4", "id" : 273137370 } }
{ "_id" : { "$oid" : "50f82aeab4aa879861000003" }, "text" : "I'm pretty how I've felt during the Golden Globes is how most of my followers feel during a NASCAR race/football game. #TweetOverload", "created_at" : { "$date" : 1358137507000 }, "id" : 290675892311715840, "user" : { "screen_name" : "JordanTMcGraw", "id" : 25172777 } }
{ "_id" : { "$oid" : "50f82aeab4aa879861000004" }, "text" : "RT @LaAbuela961: Hay personas que solo compran ropa en Jacarandas pero se pasaron criticando los vestuarios de los famosos en los Golden Globes. #NoMeJodan", "created_at" : { "$date" : 1358137507000 }, "id" : 290675892278157313, "user" : { "screen_name" : "silvita_hdez", "id" : 269487031 } }

What I want to do is parse each of those entries into my “tweets” model, which in the schema looks like this:

create_table "tweets", :force => true do |t|
  t.integer  "tid"
  t.datetime "created_at", :null => false
  t.text     "text"
  t.string   "user_name"
  t.integer  "user_id"
  t.datetime "updated_at", :null => false
end

The data from the twitter JSON should go into my table as follows:

• _id.user.screen_name -> Tweet.user_name
• _id.user.id -> Tweet.user_id
• _id.text -> Tweet.text
• _id.created_at -> Tweet.created_at
• _id.id -> Tweet.tid
  

---

The rake task I have written so far looks like this:

require 'multi_json'

namespace :db do
  task :import_tweets => :environment do
    File.open('/omittedPath/5tweets.json', 'r') do |file|
      file.each do |line|
        attrs = JSON.parse line
        twt = Tweet.find_or_initialize_by_identifier(attrs[2])
        twt.save!
      end
    end
  end
end

When I tried to parse the twitter json in the rails console, I ran the command:

JSON.parse { "_id" : { "$oid" : "50f840c2b4aa879ae4000d5d" }, "text" : "Nanananana BATMAN! Christian Bale is presenting now. #GoldenGlobes", "created_at" : { "$date" : 1358134191000 }, "id" : 290661983240454144, "user" : { "screen_name" : "egbrown27", "id" : 377696330 } }

but the response I got was

SyntaxError: (irb):11: syntax error, unexpected ':', expecting '}'
JSON.parse { "_id" : { "$oid" : "50f840c2b4aa879a...
                ^
(irb):11: syntax error, unexpected ':', expecting tASSOC
...JSON.parse { "_id" : { "$oid" : "50f840c2b4aa879ae4000d5d" }...
...                               ^
(irb):11: syntax error, unexpected ',', expecting $end
...: "50f840c2b4aa879ae4000d5d" }, "text" : "Nanananana BATMAN!...

Any help would be greatly appreciated. Thank you!

Answer 1

You should pass a string to JSON.parse. Try this:

JSON.parse '{ "_id" : { "$oid" : "50f840c2b4aa879ae4000d5d" }, "text" : "Nanananana BATMAN! Christian Bale is presenting now. #GoldenGlobes", "created_at" : { "$date" : 1358134191000 }, "id" : 290661983240454144, "user" : { "screen_name" : "egbrown27", "id" : 377696330 } }'

It will return a parsed hash:

{"_id"=>{"$oid"=>"50f840c2b4aa879ae4000d5d"}, "text"=>"Nanananana BATMAN! Christian Bale is presenting now. #GoldenGlobes", "created_at"=>{"$date"=>1358134191000}, "id"=>290661983240454144, "user"=>{"screen_name"=>"egbrown27", "id"=>377696330}}