Question

Goal

I want to simulate with a request to a HTML-document just the way a browser does. This means I not only want to download the main HTML file but also linked stuff like CSS, JS, images.

For now I only want to parse the first HTML document. I.e. I am not taking requests into account which result from parsing say the CSS (background-images, web-fonts) or JavaScript (Ajax) etc.

To implement this I need to know how a browser exactly processes a website. I haven’t found a good reference on that. Any help on that would be greatly appreciated and likely solve my problem.

Assumption

As I lacked any good references I assumed the process (without taking redirects, the rendering etc. into account) works like that:

1. A persistent HTTP-connection is established with www.facebook.com
2. The Path “/” is requested and the HTML is received
3. When the the document is fully (?) received it is parsed and a list of URLs that need to be requested is filled (divided into head and body objects?!)
4. The first URL is taken from the head-list and it is checked whether a persistent HTTP-connection is established with that host
   • If there is not, it is established and the afterwards the Object is requested
   • If there is, it is added to the connection’s “download queue”
5. Step 4 is repeated until the list is empty
6. Then Steps 4-5 are repeated for the “body list”

Is this even close to how a browser works?

Bonus question: Does the order of JavaScript and CSS files in the header make a difference?

Doubts

I conducted a few tests with the Chrome Developer Tools (Chrome 18) to confirm. I connected to www.facebook.com measuring when each Object was loaded. When reloading using Ctrl + R the results looked like this:

What puzzles me the most is that most requests are concurrent to others even when from the same host (static.ak.fbcdn.net). Pipelining is disabled my browser (which is the default setting) so why are the request still seem to happen simultaneously?

Answer 1

Browsers do use multiple connections, in order to speed up the downloading (parallel downloading of resources). They limit however the number of connections to the same host, which is one of the reasons for the existence of content-delivery networks.

The order of CSS and Script files in the header do matter, as scripts block parallel downloading (unless the script is not defered).

Also browsers parse HTML while they receive it (again to speed up things) – this is a reason if you put a script in the head that tries to manipulate DOM elements not yet loaded, you’ll get an error.

But all of this are browser implementation details that may not be important for your task.
Best – look at the source code of some headless browser to find what’s going on.