Question

I have made a simple download from http function as below (error handling is omitted for simplifcation):

function download(url, tempFilepath, filepath, callback) {
    var tempFile = fs.createWriteStream(tempFilepath);
    http.request(url, function(res) {
        res.on('data', function(chunk) {
            tempFile.write(chunk);
        }).on('end', function() {
            tempFile.end();
            fs.renameSync(tempFile.path, filepath);
            return callback(filepath);
        })
    });
}

However, as I call download() tens of times asynchronously, it seldom reports error on fs.renameSync complaining it cannot find file at tempFile.path.

Error: ENOENT, no such file or directory 'xxx'

I used the same list of urls to test it, and it failed about 30% of time. The same list of urls worked when downloaded one by one.

Testing some more, I found out that the following code

fs.createWriteStream('anypath');
console.log(fs.exist('anypath'));
console.log(fs.exist('anypath'));
console.log(fs.exist('anypath'));

does not always print true, but sometimes the first answer prints false.

I am suspecting that too many asynchronous fs.createWriteStream calls cannot guarantee the file creation. Is this true? Are there any methods to guarantee file creation?

Answer 1

You shouldn’t call write on your tempFile write stream until you’ve received the 'open' event from the stream. The file won’t exist until you see that event.

For your function:

function download(url, tempFilepath, filepath, callback) {
    var tempFile = fs.createWriteStream(tempFilepath);
    tempFile.on('open', function(fd) {
        http.request(url, function(res) {
            res.on('data', function(chunk) {
                tempFile.write(chunk);
            }).on('end', function() {
                tempFile.end();
                fs.renameSync(tempFile.path, filepath);
                return callback(filepath);
            });
        });
    });
}

For your test:

var ws = fs.createWriteStream('anypath');
ws.on('open', function(fd) {
    console.log(fs.existsSync('anypath'));
    console.log(fs.existsSync('anypath'));
    console.log(fs.existsSync('anypath'));
});