Question

I’m very new to coding in general, so I apologize ahead of time if this question should be rather obvious. Here’s what I’m looking to do, and following that I’ll post the code I’ve used so far.

I’m trying to get gzip’d csv rank data from a website and store it into a database, for a clan website that I’m working on developing. Once I get this figured out, I’ll need to grab the data once every 5 minutes. The grabbing the csv data I’ve been able to accomplish, although it stores it into a text file and I need to store it into mongodb.

Here’s my code:

var DB        =    require('../modules/db-settings.js');
var http      =    require('http');
var zlib      =    require('zlib');
var fs        =    require('fs');
var mongoose  =    require('mongoose');
var db          =   mongoose.createConnection(DB.host, DB.database, DB.port, {user: DB.user, pass: DB.password});

var request = http.get({ host: 'www.earthempires.com',
                     path: '/ranks_feed?apicode=myapicode',
                     port: 80,
                     headers: { 'accept-encoding': 'gzip' } });
request.on('response', function(response) {
  var output = fs.createWriteStream('./output');

  switch (response.headers['content-encoding']) {
    // or, just use zlib.createUnzip() to handle both cases
    case 'gzip':
      response.pipe(zlib.createGunzip()).pipe(output);
      break;
    default:
      response.pipe(output);
      break;
  }
});

db.on('error', console.error.bind(console, 'connection error:'));
db.once('open', function callback () {
  var rankSchema = new mongoose.Schema({
    serverid: Number,
    resetid: Number,
    rank: Number,
    countryNumber: Number,
    name: String,
    land: Number,
    networth: Number,
    tag: String,
    gov: String,
    gdi: Boolean,
    protection: Boolean,
    vacation: Boolean,
    alive: Boolean,
    deleted: Boolean
  })
});

Here’s an example of what the csv will look like(first 5 lines of file):

9,386,1,451,Super Kancheong Style,22586,318793803,LaF,D,1,0,0,1,0
9,386,2,119,Storm of Swords,25365,293053897,LaF,D,1,0,0,1,0
9,386,3,33,eug gave it to mak gangnam style,43501,212637806,LaF,H,1,0,0,1,0
9,386,4,128,Justpickupgirlsdotcom,22628,201606479,LaF,H,1,0,0,1,0
9,386,5,300,One and Done,22100,196130870,LaF,H,1,0,0,1,0

Answer 1

Hope it’s not too late to help, but here’s what I’d do:

1. Request the CSV formatted data and store it in memory or a file.
2. Parse the CSV data to convert each row into an object.
3. For each object, use Model.create() to create your new entry.

First, you need to create a model from your Schema:

var Rank = db.model('Rank', rankSchema);

Then you can parse your block of CSV text (whether you read it from a file or do it directly after your request is up to you.) I created my own bogus data variable since I don’t have access to the api, but as long as your data is a newline delimited block of CSV text this should work:

/* Data is just a block of CSV formatted text. This can be read from a file                                                                                                  
   or retrieved right in the response. */                                                                                                                                    
var data = '' +                                                                                                                                                              
    '9,386,1,451,Super Kancheong Style,22586,318793803,LaF,D,1,0,0,1,0\n' +                                                                                                  
    '9,386,2,119,Storm of Swords,25365,293053897,LaF,D,1,0,0,1,0\n' +                                                                                                        
    '9,386,3,33,eug gave it to mak gangnam style,43501,212637806,LaF,H,1,0,0,1,0\n' +                                                                                        
    '9,386,4,128,Justpickupgirlsdotcom,22628,201606479,LaF,H,1,0,0,1,0\n' +                                                                                                  
    '9,386,5,300,One and Done,22100,196130870,LaF,H,1,0,0,1,0\n';                                                                                                            

data = data.split('\n');                                                                                                                                                     

data.forEach(function(line) {                                                                                                                                                
    line = line.split(',');   

    if (line.length != 14)
        return;                                                                                                                                               

    /* Create an object representation of our CSV data. */                                                                                                                   
    var new_rank = {                                                                                                                                                         
        serverid: line[0],                                                                                                                                                   
        resetid: line[1],                                                                                                                                                    
        rank: line[2],                                                                                                                                                       
        countryNumber: line[3],                                                                                                                                              
        name: line[4],                                                                                                                                                       
        land: line[5],                                                                                                                                                       
        networth: line[6],                                                                                                                                                   
        tag: line[7],                                                                                                                                                        
        gov: line[8],                                                                                                                                                        
        gdi: line[9],                                                                                                                                                        
        protection: line[10],                                                                                                                                                
        vacation: line[11],                                                                                                                                                  
        alive: line[12],                                                                                                                                                     
        deleted: line[13]                                                                                                                                                    
    };                                                                                                                                                                       

    /* Store the new entry in MongoDB. */                                                                                                                                    
    Rank.create(new_rank, function(err, rank) {                                                                                                                            
        console.log('Created new rank!', rank);                                                                                                                              
    });                                                                                                                                                                      
});

You could put this in a script and run it every 5-minutes using a cron job. On my Mac, I’d edit my cron file with crontab -e, and I’d setup a job with a line like this:

*/5 * * * * /path/to/node /path/to/script.js > /dev/null