I am fetching current data from another company’s web feed. It is a simple fetch of an XML file over HTTP. They haven’t provided me with much documentation – just a URL.
Because I need to know as soon as possible when the data changes on their site, I need to poll frequently, which isn’t a satisfactory solution for either side.
I was about to recommend to them that they set-up some sort of server push – presumably a long-term HTTP connection with asynchronous updates being sent by the server. I am not very familiar with any common protocols for this. It occurred to me that they may already offer this, and I have been too ignorant to realise.
Is there a common web-based protocol for server pushes over HTTP? If there is, is there a quick way I can check if they support it before I make myself look foolish by asking for something that is already available.
(Bonus points for a platform-independent, Python-based solution, but I will take what I can get.)
What you want is HTTP Streaming; read this page. “Comet” is what this technology is commonly called. One implementation is the Ajax Push Engine (APE); the page I just gave you has several others.
Now I don’t think it’s possible to automatically test if a server supports a push technology because as of now there are no standards on this and the protocols used will vary depending on the implementation.
Alternatively you can use periodic refresh (“polling”), and the advantages of this technique are: you don’t need additional software on the server, and this can be done without the cooperation of the server you are polling (it is unfeasible to use Comet if the server you are querying won’t install it).
For more information and tricks to reduce bandwidth usage on polling, see this page. Some of these will require some effort from the server you are polling.