Question

I’m using cURL to connect to a server managed by a company called Gnip. (www.gnip.com)
Ultimately we want to consume a pipeline json feed indefinitely.

Originally when I setup our software, there was a nice little class written to maintain the connection; it was provided from socialping via gnip.

Gnip changed how they require the connection be done, so that class broke.

I can connect to the server just fine. Sometimes it stays open for days, sometimes the connection dies in seconds.

The way everything SHOULD work is: I connect to gnip and maintain an open connection. gnip sends data back to me live (as they receive it) as a json string. If no data has been sent within 30 seconds they send a ‘keep alive’ signal to let my script know it’s still connected.

Ideally the script would only disconnect when one of the two servers is shutdown. I’ve got that handled on my end via a cronjob.

The problem is that the connection sometimes closes unexpectedly. I contacted gnip and their logs say that the disconnects are not their fault.

This is all out of my normal realm. I’m sure that curl is sending some sort of error, but I don’t know how to find it in order to log it.

Here is a copy of the code I have written so far: http://pastebin.com/jpHzvbTF

I’d love a direct ‘here is how you fix it’, but I’d also love to know some terms to read up on that might lead me to my own solution.

I’ve read Keep-alive in curl / php but I found it not be as related to this as the time suggested.

Answer 1

I work for a company that is a customer of Gnip and uses the same product that you are using. Our code is in Java, rather than PHP, so I may not be able to be of great assistance to you, but here is what I have found in the time that I’ve been working with these feeds:

1. Streaming HTTP ain’t all it’s cracked up to be. A lot can go wrong between where you are and where Gnip’s endpoint is hosted.
2. You are going to need to build logic to detect disconnections and try to reconnect. Again, I’m not sure how you would do this using cURL and PHP. In Java, what has worked for us is read timeouts on the input stream and connection timeouts to force exceptions, whereupon we drop the connection and try again, but you have to be careful with these as well – too brief a TCP read timeout will see you constantly reconnecting which yields very strange behavior in Gnip’s UI. However, using something like this will allow you to catch the state where Gnip has failed to send their keep-alive newline and appropriately cycle your connection.
3. Gnip does periodic updates to their software and states this in their terms. During these updates, they may (will) drop your connection and you will be expected to reconnect. Unless there are bugs on their end, this drop will usually signal correctly and not leave your connection in a bad state, so whatever you’re using to detect a dropped connection can fire, you can reconnect, and all is well.

I wish I could give you better advice in terms of how to handle the problems you’re seeing with the specific technologies you’re using. Dig into Streaming HTTP (or Keep-Alive HTTP sessions) a little bit, see if that doesn’t get you anywhere. Definitely figure out how to trap disconnections of ANY flavor and then reconnect.

Gnip has begun advising people to implement reconnection backoff logic, which would mean that your reconnections would, say, start immediately and upon each successive failure to reconnect, wait for n*2 < 10 (say) seconds, where n is the number of connection attempts thus far, before retrying. Twitter themselves demand this as part of their streaming service, whereas Gnip merely suggests it (it is a paid service, after all), but if you want to keep your Gnip UI from getting cluttered with failed attempts, I would recommend it.

For the most part, my experience with Gnip has been pretty good. But Streaming HTTP is a highly imperfect technology (as we’ve come to find out). There’s this somewhat naive notion that you get to connect once and live happily ever after. I, too, thought that would be the case when I started, and now I’m a bit cynical. If I had my druthers, I would never support a production system built on top of Streaming HTTP with a service that is outside of my own network. I would much rather get FTP drops, for all that that is a pain in the ass at the kinds of volumes you might be talking about. Unfortunately, they’re not offered for that product line.

Good luck.