Question

Our issue is that our project has files being downloaded using wget to the file system. We are using ruby to read the downloaded files for data.
How is it possible to tell if the file is completely downloaded so we don’t read a half complete file?

Answer 1

I asked a very similar question and got some good answers… in summary, use some combination of one or more of the following:

• download the file to a holding area and finally copy to your input directory;
• use a marker file, created once the download completes, to signal readiness;
• poll the file twice and see if its size has stopped increasing;
• check the file’s permissions (some download processes block reads during download);
• use another method like in-thread download by the Ruby process.

To quote Martin Cowie, ‘This is a middleware problem as old as the hills’…