Question

In my server process, it looks like this:

Main backend processes:

1. Processes Huge list of files and , record them inside MySQL.

2. On every 500 files done, it writes “Progress Report” to a separate file /var/run/progress.log like this “200/5000 files done”

3. It is multi-processed with 4 children, each made sure to run on a separate file.

Web server process:

1. Read the output of /var/run/progress.log every 10 seconds via Ajax and report to progress bar.

When processing a very large list of files (e.g. over 3 GB archive), the processes lock up after about 2 hours of processing.

I can’t find what is going on. Does that mean that /var/run/progress.log caused an I/O deadlock?

Answer 1

In python on Linux this should not block, however try using the os module

os.open(file, os.O_NONBLOCK | os.O_RDONLY)

and make sure you close the file fh.close() as Python is a bit lazy clean up files.

http://docs.python.org/library/os.html