Question

How are files downloaded from servers in programming languages like C? I understand higher level languages have magic functions like “download_file_from_url()” but they don’t help me understand what is actually going on. I’m a little familiar with sockets but network programming in general is still a black box to me. Thanks for any help.

Answer 1

Basically, at a low-ish level, the program is opening a socket to port 80 (usually) on the server and sending it a request that looks something like this:

GET /index.html HTTP/1.1
Host: stackoverflow.com

…followed by a blank line.

The server then responds with the data, which typically consists of a few header lines, a blank line, and the requested resource. With HTTP 1.1 the default is to keep the connection alive for subsequent requests (although the server could terminate it if it liked); if I’d used HTTP 1.0 or added a Connection: close header, the server would break the connnection after sending the resource.

Check out the Wikipedia article on HTTP for details, or if you really want to get into it, check out the spec (all-in-one-page here). You can see what this looks like for yourself if you have telnet (and you probably do). Just type telnet stackoverflow.com 80 and then type in the lines above. Remember to press Enter on the blank line.

You do not want to reinvent this wheel. Virtually all languages and environments have a library available to help you that deals with all of the intricacies. (For instance, try the example above with www.stackoverflow.com instead of stackoverflow.com in both places — you get back a “moved permanently” response because the SO team want SO to be at stackoverflow.com, not www.stackoverflow.com. There are also “moved temporarily” responses, etc., etc.)