Question

I am writing a program to crawl the websites. The crawl function is a recursive one and may consume more time to complete, So I used Multi Threading to perform the crawl for multiple websites.
What exactly I need is, after completion crawling one website it call next one (which should be in Queqe) instead multiple websites crawling at a time.
I am using C# and ASP.NET.

Answer 1

The standard practice for doing this is to use a blocking queue. If you are using .NET 4.0 then you can take advantage of the BlockingCollection class otherwise you can use Stephen Toub’s implementation.

What you will do is spin up as many worker threads as you feel necessary and have them go around in an infinite loop dequeueing items as they appear in the queue. Your main thread will be enqueueing the item. A blocking queue is designed to wait/block on the dequeue operation until an item becomes available.

public class Program
{
  private static BlockingQueue<string> m_Queue = new BlockingQueue<string>();

  public static void Main()
  {
    var thread1 = new Thread(Process);
    var thread2 = new Thread(Process);
    thread1.Start();
    thread2.Start();
    while (true)
    {
      string url = GetNextUrl();
      m_Queue.Enqueue(url);
    }
  }

  public static void Process()
  {
    while (true)
    {
      string url = m_Queue.Dequeue();
      // Do whatever with the url here.
    }
  }
}