Home/ Questions/Q 1066991
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-16T20:05:37+00:00

I wrote a crawler in python, fetched urls has different types: it can be

  • 0

I wrote a crawler in python, fetched urls has different types: it can be url with html and url with image or big archives or other files. So i need fast determine this case to prevent of reading of big files such as big archives and continue crawling. How is the best way to determine url type at start of page loading?
i understand what i can do it by url name (end’s with .rar .jpg etc) but i think it’s not full solution. I need check header or something like that for this? also i need some page size predicition to prevent of large downloads. In other words set limit of downloaded page size, to prevent fast memory eating.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    If you use a HTTP HEAD request on the resource, you will get relevant metadata on the resource without the resource data itself. Specifically, the content-length and content-type headers will be of interest.

    E.g.

    HEAD /stackoverflow/img/favicon.ico HTTP/1.1
host: sstatic.net

HTTP/1.1 200 OK
Cache-Control: max-age=604800
Content-Length: 1150
Content-Type: image/x-icon
Last-Modified: Mon, 02 Aug 2010 06:04:04 GMT
Accept-Ranges: bytes
ETag: "2187d82832cb1:0"
X-Powered-By: ASP.NET
Date: Sun, 12 Sep 2010 13:38:36 GMT

    You can do this in python using httplib:

    >>> import httplib
>>> conn = httplib.HTTPConnection("sstatic.net")
>>> conn.request("HEAD", "/stackoverflow/img/favicon.ico")
>>> res = conn.getresponse()
>>> print res.getheaders()
[('content-length', '1150'), ('x-powered-by', 'ASP.NET'), ('accept-ranges', 'bytes'), ('last-modified', 'Mon, 02 Aug 2010 06:04:04 GMT'), ('etag', '"2187d82832cb1:0"'), ('cache-control', 'max-age=604800'), ('date', 'Sun, 12 Sep 2010 13:39:26 GMT'), ('content-type', 'image/x-icon')]

    This tells you it’s an image (image/* mime-type) of 1150 bytes. Enough information for you to decide if you want to fetch the full resource.

    Additionally, this header tells you the server accepts HTTP partial content request (accept-ranges header) which allows you to retrieve data in batches.

    You will get the same header information if you do a GET directly, but this will also start sending the resource data in the body of the response, something you want to avoid.

    If you want to learn more about HTTP headers and their meaning, you can use an online tool such as ‘Fetch’

    • 0