Question

I’m writing a download manager, and I’ve noticed that all the web pages I’ve encountered don’t seem to set the ContentLength header, whereas other media types (images) do. However, when I load a page (in Firefox), I get a progress as it loads. What’s that based off if not file size?

Answer 1

Because a web page may be the sum of many downloads, any of which may have an unknown size, the progress is more the number of items completed compared to the number of items requested than a precise measure of bytes. And I think most browsers include dns resolution as a significant chunk of the progress.

First choose an amount of progress to give to dns, redirects and other handshake involved in getting the first byte of the main document, say 20% of total. When you get the first byte of the main document, you are at 20% progress.

You might also reserve a little tail space of the progress for running javascripts and doing page layout. Say another 10% leaving 70% to represent downloads.

As the main document is downloading, count all the additional resources that will need to be downloaded. If there are 50 additional resources, then the main document represent 2% of the remaining 70% of the download, so when it finishes you are at almost 22% progress regardless of how big it was. As each of those 50 resources arrives the progress goes up a little.

You might weight things as the size becomes known or even by type. So maybe css files count as 1 item, images as 3 and movies as 10. For any of these items that the size is given, you can increment the overall progress more precisely.

In a nutshell, it is an aggregate progress bar representing any number of tasks. Even if the number of bytes for a given task is known the time is not.