Question

I am working on a website that allows people to upload their images. It’s mainly just a learning experience, and I’m writing it in python. One thing it should be able to do is take a url where an image resides, then reupload that to its own server. So here’s the question:

One person gives my website a url, then my server starts downloading the image. While it is downloading that image, another request comes in. Is there any way to get the server to give a response (Like saying the server is busy) even when it’s doing something else? Is there a better way to go about this?

Thanks in advance!

Answer 1

I wrote up an example that I hope you find helpful. It consists of a urls.txt file that lists the jobs to process and a xdownloader.py file which downloads using multiple processes. I tested it on a Linux machine running Python 2.7.

xdownloader.py

from multiprocessing import Process
import urllib2

def main():

  try:

    # Read URLs/Destinations from a file
    jobs = []
    with open("urls.txt","r") as ifile:
      for line in ifile:
        jobs.append(line.split(" "))

    # Create a process list to keep track of running processes
    process_list = []
    # Iterate through our jobs list
    for url, save_to in jobs:
      # Create a new process that runs function 'download'
      p = Process(target=download, args=(url, save_to))
      # Save it to a list
      process_list.append(p)
      # Start the process
      p.start()
  except KeyboardInterrupt:
    print("Received keyboard interrupt (ctrl+c). Exiting...")
  finally:
    # Wait for all processes to finish before exiting
    for process in process_list:
       # Wait for this process to finish
       process.join()
    print("All processes finished successfully!")


def download(url, destination):
  # Open a request
  request = urllib2.urlopen(url)
  # Read and save the webpage data to a file
  with open(destination, "w+") as save_file:
    print("Downloading {0}".format(url))
    save_file.write(request.read())

if __name__=="__main__":
  main()

urls.txt

http://google.com google.html
http://yahoo.com yahoo.html
http://news.google.com news.google.html
http://reddit.com reddit.html
http://news.ycombinator.com news.ycombinator.html

Running python xdownloader.py I get:

mike@localhost ~ $ python xdownloader.py 
Downloading http://news.ycombinator.com
Downloading http://reddit.com
Downloading http://news.google.com
Downloading http://google.com
Done downloading http://google.com
Done downloading http://news.ycombinator.com
Done downloading http://reddit.com
Downloading http://yahoo.com
Done downloading http://news.google.com
Done downloading http://yahoo.com

You can see that the jobs are running asynchronously… Some jobs start earlier, but finish later than the others. (I’m looking at you news.google.com!) If this example doesn’t fit your needs, please let me know in the comments.