Question

Is there a easy and reliable way to confirm that a web download completed successfully to download using Python or WGET [for large files]? I want to make sure the file downloaded in its entirety before performing another action.

Answer 1

Given many (most in practice, I believe) HTTP/1.1 header sections, you can get an expectation about how long the entity body is. If you have that expectation, you can decide if you got all the entity data. See RFC 2616 section 4.4 for full details, but essentially:

• sometimes the content-length accurately reflects the length of the entity body
• sometimes there can be no entity body, depending on the response code or if the response is responding to a HEAD request
• sometimes the request is transfer encoded; and there is some marker in the HTTP data which says ‘I’m done now’ (Transfer-Encoding: chunked)
• and sometimes, the message is officially done when the connection closes (in which case, you cannot differentiate between getting the whole thing and being cut off early)

In all cases but the last one, you can tell if you’ve got the whole thing or not. I don’t know if any tool in particular (wget or an existing python library) gives you an easily interpretable signal that your response was or wasn’t truncated.