Question

I have to extract the data from fxstreet
I extracted the HTML code with firebug and it seems like the webpage is using Comet and HTTP streaming.

I would like to fill in a dictionary every second with the data without having to refresh the page. I did it with urllib.urlopen but I am obliged to do a query every second.

Someone knows a proper way to pull the data from the Comet & http streaming?
thanks

Answer 1

You’ll probably want to use gevent, Tornado, or Twisted to write an asynchronous HTTP client to consume your service. There have been quite a few projects built for working with the Twitter Streaming API that you might look to for inspiration:

• https://github.com/fiorix/twisted-twitter-stream – twisted
• https://github.com/dustin/twitty-twister – twisted
• https://github.com/atl/twitstream – asyncore, pycurl or tornado
• https://github.com/godavemon/TwitTornado – tornado
• https://github.com/thruflo/close.consumer – gevent