Home/ Questions/Q 8429183
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-10T05:15:52+00:00

I’m having a bit of trouble figuring this out so if you can help

  • 0

I’m having a bit of trouble figuring this out so if you can help that would be great.

I have written a bit of code that gets a list of URLs from a sitemap. I then go each page, scrape all the links, and then test their status(200,404, etc).

I am using HttpClient. I have it all working OK but as I am new to Java I reckon my code is a complete hack/maze and I could most likely get far better performance if it was organised correctly. So what I have is

Main class – This builds the gui

Parse the sitemap class – This parses the sitemap and get a list of the urls.

Class called PingURl – I’m sure my above is poor but this is the bit I reckon is worst. This class opens all the urls, scrapes them for links, then tests the links for their status and returns it. I presume this class should be broken down? Most importantly I think I should be isolating the testing of the links in it’s own class, so it would be easy to implement threads later on?

Basically I’m looking for advice. If someone could help me with laying out the project a bit better. Secondly I believe this is my weakest area so to improve I need to learn more about this, I don’t even know what to call this(design/layout problem?). Can you also recommend resources to learn more about this?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Java is a language which IMO, embodies good OO design. Designing with OO in mind is very effective.

    http://java.sun.com/developer/onlineTraining/Programming/BasicJava2/oo.html

    In terms of your problem, I think it works pretty well. You are kind of following a Model-View-Controller pattern: http://www.codinghorror.com/blog/2008/05/understanding-model-view-controller.html

    I can’t tell you how to design your code, but I will tell you how I would do it if I were presented with the same problem:

    I would get a class to represent a hyperlink. In that class there would be the hyperlink that has been scraped, and a getter and setter. As well as a boolean value and a ping function. That means that if I create a “hyperlink” object, then I can invoke (“ping”) on that object because it is part of that class. That means that the ping function simply sets the boolean that represents connectivity.

    That way, your parser basically gets a page, and for each link it finds, it creates a new “hyperlink” object based on your own “hyperlink” class. And puts it in an array. So once your parser executes you get an array of pointers to hyperlinks.

    Then all you have to do is invoke the ping function on each one to see if it is there.

    I think this design is the best because it scales from doing one hyperlink test to doing 1000 quite easily.

    There is soooo much stuff out there about design, there are countless principals as well. There is never one absolute way to solve a problem. But the more experience you get, and the more you read up on design patterns and models the better you will get at it 🙂

    • 0