Question

I’m trying to scrape data from an external JSON feed and store it in my mongoDB.

request(url, function (error, response, body) {
    if (!error && response.statusCode == 200) {
        var jsonObj = JSON.parse(body);
        // console.log(jsonObj.events[1].id) <-- this works
        for (var i = 0; i < jsonObj.events.length; i++) { 
            // add jsonObj.events[i] as a new record to table
            // console.log(jsonObj.events[i].id) <-- this works
            Wnet.findOne({ id : jsonObj.events[i].id }, function (err, doc){
                if (err || doc == null) {
                     // console.log(jsonObj.events[i].id) <-- this doesn't work!
                     // record is new, add it
                }
            });
         }
     }
});

Notice the console.log statements… the last one (inside the .findOne‘s callback) doesn’t work.

I’m still learning the ins and outs of JavaScript, but I believe this is a scope issue… the callback function doesn’t know jsonObj. But, I don’t know how to pass it into the function.

Taking a step back, I’m trying to check the collection for existence of the record. If it can’t find it, I want to add the record from the fetched JSON array. If there’s a better way to do that, I’d love your help in finding it.

Answer 1

This is not a scope issue. In terms of scope, it will look for the variable by bubbling up. So in the findOne it has the variable jsonObj. It will bubble up to the var jsonObj = ... above.

I believe your issue is that you are looping through jsonObj.events and firing off findOne for each. You are then trying to console jsonObj.events[i].id in the findOne callback for each. The problem is that your i is incremented each time. And will actually end when i >= jsonObj.events.length. So if the length is 9 then i will end up being 10 when the loop exits. So your logging will not work. EDIT: To clarify, the logging will not work because within the loop all the findOne will fire off but the callbacks won’t happen right away. Because the callbacks are async. So by the time the callbacks happen the loop is likely finished, and it will try to log an events that does not exist.

Try this:

request(url, function (error, response, body) {
    if (!error && response.statusCode == 200) {
        var jsonObj = JSON.parse(body);
        //console.log(jsonObj.events[1].id) <-- this works
        for (var i = 0; i < jsonObj.events.length; i++) { 
            // add jsonObj.events[i] as a new record to table
            //console.log(jsonObj.events[i].id) <-- this works
            Wnet.findOne({ id : jsonObj.events[i].id }, function (err, doc){
                if (err || doc == null) {
                    console.log(jsonObj.events[this].id);
                    //record is new, add it
                }
            }.bind(i));
         }
     }
});

This is just one way of dealing with it. If the bind is confusing then try a different way. You could also do something like:

    request(url, function (error, response, body) {
    if (!error && response.statusCode == 200) {
        var jsonObj = JSON.parse(body);
        //console.log(jsonObj.events[1].id) <-- this works
        for (var i = 0; i < jsonObj.events.length; i++) { 
            // add jsonObj.events[i] as a new record to table
            //console.log(jsonObj.events[i].id) <-- this works
            doFindOne(jsonObj.events, i);
         }
     }
});


function doFindOne(events, i) {
    Wnet.findOne({ id : events[i].id }, function (err, doc){
        if (err || doc == null) {
            console.log(events[i].id);
            //record is new, add it
        }
    });
}

In this second example we are calling a function which will keep the i in scope to that call. This may make more sense. (binding tends to confuse more than clarify)