Question

In another question on stackoverflow I got a hint that I can use thread pools for the producer-consumer pattern my crawlers are creating.

However, I just cannot find out how to implement it.

In a producer consumer thread on SO they just use the producer consumer to manage the producers and consumers (which in my case would be the crawler themselves; and this is not so much different from my for-loop), but this does not seem the intention of the commentor in my article (as he could not see I used a for loop). The workload is still shared via a queue there.

I also thought about passing a Website object to ExecutorService.submit() with this implementation (and remove Runnable from Crawler):

public class Website implements Runnable {
    private URL url;

    public Website(URL url) {
        this.url = url;
    }

    @Override
    public void run() {
        Crawler crawler = new Crawler();
        crawler.crawl(url);
    }
}

But the problem is that

1. I think there are too many crawlers being generated
2. Crawler() expects a queue of already visited websites

How can I properly implement the producer, consumer pattern in my crawler problem? I’m getting totally confused about it all.
I checked so many websites about it on the web and all seem to use it differently.

Answer 1

I think I will need to see more code in order to understand.
However, what you can do in order to have producer-consumer is have your Crawler class be a consumer,
and have the code the uses the executor as a consumer.
The Crawler will take objects of WebSite from a queue or other shared data structure (synchronized data structure) between the producer and the consumer.
What you should ask yourself when selecting the data structure is some of the following questions –
A. Are there priorities among the sites to be crawled?
If so – consider using a PriorityBlockingQueue.

B. Is the crawling order important but all priorities are the same?
If so – consider using a LinkedBlockingQueue

C. Can you categorize the links somehow?
If so , maybe you can have several shared data structures, with a map of categories to them.

I am sure you can come up with many ideas on how to build this shared data structure on you own, these were just my thoughts.

To conclude –
1. Have Crawler extend Runnable
2. Have Crawler extract a “Job” (WebSite class) from a shared data structure (i.e – blocking queue).
3. Have producer put a job to the shared data structure before using the executor.