Question

I am writing a Python/Django application which transfer files from server to the local machine using rsync protocol. We will be dealing with the large files so the progress bar is mandatory. --progress argument in rsync command does this beautifully. All the detail progresses are shown in the terminal. How can I show that progress in web browser? Is there any hook function or something like that? Or Can I store the progress in a log file, call it and update it every one minute or so?

Answer 1

The basic principle is to run rsync in subprocess, expose a web API and get updates via javascript

Here’s an example.

import subprocess
import re
import sys

print('Dry run:')
cmd = 'rsync -az --stats --dry-run ' + sys.argv[1] + ' ' + sys.argv[2]
proc = subprocess.Popen(cmd,
                        shell=True,
                        stdin=subprocess.PIPE,
                        stdout=subprocess.PIPE,)

remainder = proc.communicate()[0]
mn = re.findall(r'Number of files: (\d+)', remainder)
total_files = int(mn[0])
print('Number of files: ' + str(total_files))

print('Real rsync:')
cmd = 'rsync -avz  --progress ' + sys.argv[1] + ' ' + sys.argv[2]
proc = subprocess.Popen(cmd,
                        shell=True,
                        stdin=subprocess.PIPE,
                        stdout=subprocess.PIPE,)

while True:
             output = proc.stdout.readline()
if 'to-check' in output:
             m = re.findall(r'to-check=(\d+)/(\d+)', output)
             progress = (100 * (int(m[0][1]) - int(m[0][0]))) / total_files
             sys.stdout.write('\rDone: ' + str(progress) + '%')
             sys.stdout.flush()
             if int(m[0][0]) == 0:
                      break

print('\rFinished')

But this only shows us the progress in our standard output (stdout).

We can however, modify this code to return the progress as a JSON output and this output can be made available via a progress webservice/API that we create.

On the client side use, we will then write javascript (ajax) to contact our progress webservice/API from time-to-time, and using that info update something client side e.g. a text msg, width of an image, color of some div etc