Question

I am using java with the twitter search api to store tweets in mongoDB, is there a way to store the json from the twitter api as is?

I am also considering using python, is there a way doing it in python?

Answer 1

I don’t know if you are still interested in an answer to this but from a purely python standing this is how I store the raw tweet json:

import tweetstream # Needed For Twitter API Capture (Make sure using modified version with proxy support)
import argparse    # Needed for taking cmd line input
import gzip        # Needed for compressing output
import json        # Needed for Data conversion for easier DB import
import ast         # Also Needed for Data conversion

collector = argparse.ArgumentParser(description='Collect a lot of Tweets')        # This line sets up the argument collector
collector.add_argument('--username', dest='username', action="store")             # This line collects the Username
collector.add_argument('--password', dest='password', action="store")             # This line collects the password
collector.add_argument('--outputfilename', dest='outputfilename', action="store") # This line collects the output filename

args = collector.parse_args()                                                     # Setup args to store cmd line arguments

def printusername():                                                              # define the username argument

        print args.username

def printpassword():                                                              # define the password argument

        print args.password

def printoutputfilename():                                                        # define the output filename

        print args.outputfilename

output=gzip.open(args.outputfilename, "a")                                        # Open the output file for GZIP writing

with tweetstream.TweetStream(args.username, args.password) as stream:             # Open the Twitter Stream
    for tweet in stream:                                                          # For each tweet within the twitter stream
        line = str(tweet)                                                         # turn the tweet into a string
        line = ast.literal_eval(line)                                             # evaluate the python string (dictionary)
        line = json.dumps(line)                                                   # turn the python dictionary into valid JSON
        output.write(line)                                                        # write the line to the output file
        output.write("\n")  

to run this just: “python myscript.py –username yourusername –password yourpassword –outputfilename yourpathandfilename”

You’ll need tweetstream argparse gzip json and ast modules installed. All of which can be installed through pip or easy_install or most ubuntu/fedora package managers.

The output file the script will create is a simple gzip compressed text file where each line is a new json string containing the complete tweet json object. Since the script runs until you hit your rate limit, it doesn’t close the gzip file with a proper EOF. However python doesn’t care so you can open it with another script, nor does 7zip or winrar.

I hope that helps. 🙂