Question

I wrote a crawler which uses a user loaded list of words to perform multiple searches on a site and parse each resulting page.

To achieve this, I wrote a main method which loops through the list of words, sending multiple http get requests asynchronously (using ThreadPool.QueueUserWorkItem) to obtain all pages of results for the current word until no more results are found, and then moves on to the next word.

The callback function accesses a public static class with a static method that parses the returned html and acts accordingly. Right now, I have it locking on the class type each time something accesses the method:

    static class Parser
{
    public static string ResponseAsString(HttpWebResponse response)
    {
        lock (typeof(Parser))
        {
            try
            {
                Stream stream;
                if (response == null || (stream = response.GetResponseStream()) == null) return string.Empty;
                using (var sr = new StreamReader(stream))
                    return sr.ReadToEnd();
            }
            catch { return string.Empty; }
        }
    }

    public static void CallbackMethod_ParseData(string html)
    {
        lock (typeof(Parser))
        {
            //Do some work here
        }
    }
}

So down to my question: how can I more efficiently lock the class and notify waiting threads when the lock is released, or should I not make Parser a static class and instead let each ThreadPool thread instantiate its own instance of this class?

Thanks so much for your help,

-still learning

Answer 1

The locks aren’t necessary. Static methods are thread-safe as long as they aren’t sharing static data, which according to your example, they aren’t.