Question

I am trying to crawl 300,000 URLs. However, somewhere in the middle, the code hangs when trying to retrieve the response code from a URL. I am not sure what is going wrong since a connection is being established but the problem is occurring after that. I have modified the code setting the read time out and the request property as suggested.However, even now the code is unable to obtain the response code!
Any suggestions/pointers will be greatly appreciated. Also, is there any way to ping a website for a certain time period and if it’s not responding just proceed to the next one?

Here is my modified code snippet:

URL url=null;

try
{
   Thread.sleep(8000);
}
catch (InterruptedException e1)
{
   e1.printStackTrace();
}

 try
{
   //urlToBeCrawled comes from the database
   url=new URL(urlToBeCrawled);
}
catch (MalformedURLException e)
{
   e.printStackTrace();
 //The code is in a loop,so the use of continue.I apologize for putting code in the catch block.
  continue;
}
 HttpURLConnection huc=null;
 try
{
   huc = (HttpURLConnection)url.openConnection();

}
catch (IOException e)
{
   e.printStackTrace();
}
 try
 {
    //Added the request property
   huc.addRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)");
  huc.setRequestMethod("HEAD");

 }
 catch (ProtocolException e)
 {
    e.printStackTrace();
 }

 huc.setConnectTimeout(1000);
 try
 {
    huc.connect();

  }
 catch (IOException e)
 {

    e.printStackTrace();
    continue;
  }

 int responseCode=0;
 try
 {
   //Sets the read timeout
   huc.setReadTimeout(15000);
   //Code hangs here for some URL which is random in each run
   responseCode = huc.getResponseCode();

  }
 catch (IOException e)  
{
   huc.disconnect();

   e.printStackTrace();
   continue;
}
if (responseCode!=200)
{
   huc.disconnect();
   continue;
 }

Answer 1

It is hanging because the response code was never received in the byte stream. You will want to look at an http debugger and see what was actually received, if anything at all. It did however appear to open the TCP connection to the server. It might not like your user-agent (which may not have been set to what you think it was) or the request method of HEAD, or it could be a server with limited bandwidth. You could use Socket class to just open a connection and ready the bytes in manually to see what you are/aren’t recieving.

On a side note, using only Socket is not actually a bad approach depending on what you want to do. It sounds like you are writing an http server checker, in which case you will get more functionality out of using just Socket directly as you will be able to engineer better and much more optimized techniques (you are working with a large volume of low level network io after all).