Question

I’m currently researching how to add persistence to a realtime twitter json feed in node.
I’ve got my stream setup, it’s broadcasting to the client, but how do i go about storing this data in a json database such as couchdb, so i can access the stores json when the client first visits the page?

I can’t seem to get my head around couchdb.

var array = { 
    "tweet_id": tweet.id,
    "screen_name": tweet.user.screen_name,
    "text" : tweet.text, 
    "profile_image_url" : tweet.user.profile_image_url 
};

db.saveDoc('tweet', strencode(array), function(er, ok) {
    if (er) throw new Error(JSON.stringify(er));
    util.puts('Saved my first doc to the couch!');
});

db.allDocs(function(er, doc) {
    if (er) throw new Error(JSON.stringify(er));
    //client.send(JSON.stringify(doc));
    console.log(JSON.stringify(doc));
    util.puts('Fetched my new doc from couch:');
});

These are the two snippets i’m using to try and save / retrieve tweet data. The array is one individual tweet, and needs to be saved to couch each time a new tweet is received.

I don’t understand the id part of saveDoc – when i make it unique, db.allDocs only lists ID’s and not the content of each doc in the database – and when it’s not unique, it fails after the first db entry.

Can someone kindly explain the correct way to save and retrieve this type of json data to couchdb?
I basically want to to load the entire database when the client first views the page. (The database will have less than 100 entries)

Cheers.

Answer 1

• You need to insert the documents in the database. You can do this by inserting the JSON that comes from the twitter API or you can insert one status at a time (for loop)
• You should create a view that exposes that information. If you saved the JSON directly from Twitter you are going to need to emit several times in your map function
• There operations (ingestion and querying) are not the same thing, so you should really do them at the different times in your program.
• You should consider running a bg process (maybe in something as simple as a setInterval) that updates your database. Or you can use something like clarinet (http://github.com/dscape/clarinet) to parse the Twitter streaming API directly.

I’m the author of nano, and here is one of the tests that does most of what you need:

• https://github.com/dscape/nano/blob/master/tests/view/query.js

For the actual query semantics and for you learn a bit more of how CouchDB works I would suggest you read:

• http://guide.couchdb.org/editions/1/en/index.html

I you find it useful I would suggest you buy the book 🙂