Question

I am using URL.openStream() to download many html pages for a crawler that I am writing. The method runs great locally on my mac however on my schools unix server the method is extremely slow. But only when downloading the first page.

Here is the method that downloads the page:

public static String download(URL url) throws IOException {
    Long start = System.currentTimeMillis();
    InputStream is = url.openStream();
    System.out.println("\t\tCreated 'is' in "+((System.currentTimeMillis()-start)/(1000.0*60))+"minutes");
    ...
}

And the main method that invokes it:

LinkedList<URL> ll = new LinkedList<URL>();
ll.add(new URL("http://sheldonbrown.org/bicycle.html"));
ll.add(new URL("http://www.trentobike.org/nongeo/index.html"));
ll.add(new URL("http://www.trentobike.org/byauthor/index.html"));
ll.add(new URL("http://www.myra-simon.com/bike/travel/index.html"));
for (URL tmp : ll) {
    System.out.println();
    System.out.println(tmp);
    CrawlerTools.download(tmp);
}

Output locally (Note: all are fast):

http://sheldonbrown.org/bicycle.html
Created ‘is’ in 0.00475minutes

http://www.trentobike.org/nongeo/index.html
Created ‘is’ in 0.005083333333333333minutes

http://www.trentobike.org/byauthor/index.html
Created ‘is’ in 0.0023833333333333332minutes

http://www.myra-simon.com/bike/travel/index.html
Created ‘is’ in 0.00405minutes

Output on School Machine Server (Note: All are fast except the first one. The first one is slow regardless of what the first site is):

http://sheldonbrown.org/bicycle.html
Created ‘is’ in 3.2330666666666668minutes

http://www.trentobike.org/nongeo/index.html
Created ‘is’ in 0.016416666666666666minutes

http://www.trentobike.org/byauthor/index.html
Created ‘is’ in 0.0022166666666666667minutes

http://www.myra-simon.com/bike/travel/index.html
Created ‘is’ in 0.009533333333333333minutes

I am not sure if this is a Java issue (*A problem in my Java code) or a server issue. What are my options?

---

When run on the server this is the output of the time command:

real    3m11.385s
user    0m0.277s
sys     0m0.113s

I am not sure if this is relevant… What should I do to try and isolate my problem..?

Answer 1

You’ve answered your own question. It’s not a Java issue, it has to do with your school’s network or server.

I’d recommend that you report your timings in milliseconds and see if they’re repeatable. Run that test in a loop – 1,000 or 10,000 times – and keep track of all the values you get. Import them into a spreadsheet and calculate some statistics. Look at the distribution of values. You don’t know if the one data point that you have is an outlier or the mean value. I’d recommend that you do this for both networks in exactly the same way.

I’d also recommend using Fiddler or some other tool to watch network traffic as you download. You can get better insight into what’s going on and perhaps ferret out the root cause.

But it’s not Java. It’s your code, your network. If this was a bug in the JDK it would have been fixed a long time ago. Suspect yourself first, last, and always.

UPDATE:

My network admin assured me that this
was a bad java implementation Not a
network problem. What do you think?

“Assured” you? What evidence did s/he produce to support this conclusion? What data? What measurements were taken? Sounds like laziness and ignorance to me.

It certainly doesn’t explain why all the other requests behave just fine. What changed in Java between the first and subsequent calls? Did the JVM suddenly rewrite itself?

You can accept it if you want, but I’d say shame on your network admin for not being more curious. It would have been more honorable to be honest and say they didn’t know, didn’t have time, and weren’t interested.