Question

I’m currently attempting to load several text files into MongoDB (they’re in JSON format).

I tried using an OS walk, but I seem to be having trouble.
My current method is:

>>> import pymongo
>>> import os
>>> import json
>>> from pymongo import Connection
>>> connection = Connection()
>>> db = connection.Austin
>>> collection = db.tweets
>>> collection = db.tweet_collection
>>> db.tweet_collection
Collection(Database(Connection('localhost', 27017), u'Austin'), u'tweet_collection')
>>> collection
Collection(Database(Connection('localhost', 27017), u'Austin'), u'tweet_collection')
>>> tweets = db.tweets
>>> tweet = open(os.path.expanduser('~/Tweets/10_7_2012_12:09-Tweets.txt'),'r')
>>> for line in tweet:
...      d = json.loads(line)
...      tweets.insert(d)
... 

For inserting a single Tweet.
I want to be able to open multiple files and run that same piece of code, namely the for loop that turns the JSON into python dictionaries and inserts it into the collection, autonomously.

Does anyone have a solid example of how to do this, complete with an explanation?

While we’re on the topic, I’m attempting to use MongoDB with a poor understanding of databases (silly and stupid, I know), but MongoDB can support multiple instances of databases at the same time, and stores collections, which are groups of documents, and you can insert individual documents, correct?

(Also, please ignore the inconsistency between the collections tweets and tweet_collection.. I was just experimenting to get a better understanding)

Answer 1

untested

from glob import iglob
import os.path
import pymongo
import json

for fname in iglob(os.path.expanduser('~/Tweets/*.txt')):
    with open(fname) as fin:
        tweets = json.load(fin)
        for tweet in tweets:
            db.tweets.insert(tweet)

This loops over all the filenames in ‘~/Tweets/*.txt’, opens it, and loads 1 or more tweets from the file into a Python dictionary – note the use of .load instead of .loads – the difference being .load() takes a file-like object while .loads() takes a string. Then for each tweet, inserts that into the database. (Note I’ve used db.tweets.insert instead of tweets = db.tweets as I personally find the ‘db’ prefix a reminder it’s a DB op and not some other object)

As to your understanding on MongoDB re: DB’s/collections/documents – yes, you’re correct.