I’m using python feedparser in an aggregator client that runs behind a squid proxy. I want it to send a cache-control: max-age=600 header in the request, so that we get a reasonably up-to-date response. (At the moment the feeds are returned by the proxy from its cache, even days after they changed, which is reasonable based on heuristic expiry but not good enough.)
There doesn’t seem to be any direct api in feedparser to do this so what’s the best way? I don’t really want to change the source.
update: there’s a bug, 224, asking for a way to add arbitrary headers, with partial patches, but not yet merged. That’s probably the cleanest way. Otherwise it seems I need to monkeypatch either urllib or feedparser. ick.
It seems to me there are two ways:
1- wait for http://code.google.com/p/feedparser/issues/detail?id=224 to be fixed. I put up a patch that lets you send
extra_headers={'Cache-control': 'max-age=0'}and we’ll see if they accept it.2- monkeypatch in to urllib2 to put some extra headers on the request, which seems to be the only answer without changing feedparser.
Better answers very welcome…
update 2010-10-29 patch is now merged upstream, and waiting for a release