Question

Just for fun, I develop a download manager and I’d like to know if reading a large stack of data (i.e. 80 or 100KB) from a socket over the net makes the download speed higher, instead of reading 4KB for each loop iteration?

(My average download speed is 200KBPS when I download a file with firefox for example)

Thanks, Nir Tayeb.

Answer 1

The answer is NO.

your network transfer rate (200kbps) indicates that buffering 4k or 8k or 200k will hardly make a difference. The time spent between reads is too small. The bottleneck seems to be your transfer rate anyway.

Let’s try with a stackoverflow 30.9MB mp3 podcast:

NOTE: This is a unreliable hack whose results can be affected by a lot
of factors – useful for demonstration
purposes only)

import urllib2
import time

def report(start, size, text):
    total = time.time() - start
    print "%s reading took %d seconds, transfer rate %.2f KBPS" % (
            text, total, (size / 1024.0) / total)

start = time.time()
url = ('http://itc.conversationsnetwork.org/audio/download/'
       'ITC.SO-Episode69-2009.09.29.mp3')
f = urllib2.urlopen(url)
start = time.time()
data = f.read() # read all data in a single, blocking operation
report(start, len(data), 'All data')
f.close()

f = urllib2.urlopen(url)
start = time.time()
while True:
    chunk = f.read(4096) # read a chunk
    if not chunk:
        break
report(start, len(data), 'Chunked')
f.close()

The results in my system:

All data reading took 137 seconds, transfer rate 230.46 KBPS
Chunked reading took 137 seconds, transfer rate 230.49 KBPS

So for my system, 2 megabit connection, file size, server chosen, it is not much of a difference if I used chunked reading or not.