We are getting some unexpected results randomly from some servers when trying to open an InputStream from an HttpURLConnection. It seems like those servers would accept the connection and reply with a ‘stay-alive’ header which will keep the Socket open but doesn’t allow data to be sent back to the stream.
That scenario makes an attempt for a multi-threaded crawler a little ‘complicated’, because if some connection gets stuck, the thread running it would never return… denying the completion of it’s pool which derives in the controller thinking that some threads are still working.
Is there some way to read the connection response header to identify that ‘stay-alive’ answer and avoid trying to open the stream??
I’m not sure what I’m missing here but it seems to me you simply need getHeaderField()?