Question

I have a file that pulls info from twitter in python and runs the status messages into my shell. I want to take them from the shell and into a database. I do not know how to do that. I do not have a database made for it either, I will go on the database stack to question that. My code is as follows:

import time
import MySQLdb
import tweepy
from tweepy.streaming import StreamListener
from tweepy import OAuthHandler
from tweepy import Stream

# Go to http://dev.twitter.com and create an app.
# The consumer key and secret will be generated for you after
consumer_key=" # Omitted "
consumer_secret=" # Omitted "

# After the step above, you will be redirected to your app's page.
# Create an access token under the the "Your access token" section
access_token=" # Omitted"
access_token_secret=" #Omitted "

auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)

api = tweepy.API(auth)

# If the authentication was successful, you should
# see the name of the account print out
print api.me().name

class StdOutListener(StreamListener):
        """ A listener handles tweets are the received from the stream.
        This is a basic listener that just prints received tweets to stdout.
        """
        def on_data(self, data):
            print data
            return True

        def on_error(self, status):
            print status

if __name__ == '__main__':
        l = StdOutListener()

        stream = Stream(auth, l)    
        stream.filter(track=['search term'])

How do I get the info from the stream into a database? I also want to filter the stream to only allow certain information in. The only information I want to go into the database is the following:

1. Message Author
2. Message
3. Date / Time Stamp
4. GEO if available
5. Source ie Tweetdeck, web, mobile yaddda yadda
6. Was the message RT’d

Answer 1

This question asks for a lot of information, so I am just going to give you an overview of what you need…

First off, there are examples of this exact approach both on SO and across internet. Here is a paste example from this tutorial

class StreamListener(tweepy.StreamListener):

    status_wrapper = TextWrapper(width=60, initial_indent='    ', 
                                    subsequent_indent='    ')
    conn = mdb.connect('localhost', 'dbUser','dbPass','dbBase')

    def on_status(self, status):
        try:
            cursor = self.conn.cursor()
            cursor.execute('INSERT INTO tweets (text, date) VALUES (%s, NOW())' ,(status.text))
            print self.status_wrapper.fill(status.text)
            print '\n %s  %s  via %s\n' % (status.author.screen_name, status.created_at, status.source)
        except Exception, e:
            # Catch any unicode errors while printing to console
            # and just ignore them to avoid breaking application.
            pass

This example uses a different database driver. But you would use the on_status handler to receive new data, and split it up into values. Then you create an sql INSERT to put it into your database.

Here is an sqlite3 example from this SO question:

cur.execute("INSERT INTO TWEETS(?, ?, ?, ?)", (status.text, 
                                               status.author.screen_name, 
                                               status.created_at, 
                                               status.source))

Both of these example require that you use your database client to connect, and then get a cursor (which is an object that makes your queries and lets you look through the results). You can check out a tutorial on MySQLdb for a pretty good overview of how to set everything up and make queries.

If you end up having a more focused problem, then that can be addressed separately.