Question

I have WPF app that processes a lot of urls (thousands), each it sends off to it’s own thread, does some processing and stores a result in the database.

The urls can be anything, but some seem to be massively big pages, this seems to shoot the memory usage up a lot and make performance really bad. I set a timeout on the web request, so if it took longer than say 20 seconds it doesn’t bother with that url, but it seems to not make much difference.

Here’s the code section:

               HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(urlAddress.Address);
                            req.Timeout = 20000;
                            req.ReadWriteTimeout = 20000;
                            req.Method = "GET";
                            req.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;

                            using (StreamReader reader = new StreamReader(req.GetResponse().GetResponseStream()))
                            {
                                pageSource = reader.ReadToEnd();
                                req = null;
                            }

It also seems to stall/ramp up memory on reader.ReadToEnd();

I would have thought having a cut off of 20 seconds would help, is there a better method? I assume there’s not much advantage to using asynch web method as each url download is on its own thread anyway..

Thanks

Answer 1

In general, it’s recommended that you use asynchronous HttpWebRequests instead of creating your own threads. The article I’ve linked above also includes some benchmarking results.

I don’t know what you’re doing with the page source after you read the stream to end, but using string can be an issue:

System.String type is used in any .NET application. We have strings
as: names, addresses, descriptions, error messages, warnings or even
application settings. Each application has to create, compare or
format string data. Considering the immutability and the fact that any
object can be converted to a string, all the available memory can be
swallowed by a huge amount of unwanted string duplicates or unclaimed
string objects.

Some other suggestions:

• Do you have any firewall restrictions? I’ve seen a lot of issues at work where the firewall enables rate limiting and fetching pages grinds down to a halt (happens to me all the time)!
• I presume that you’re going to use the string to parse HTML, so I would recommend that you initialize your parser with the Stream instead of passing in a string containing the page source (if that’s an option).
• If you’re storing the page source in the database, then there isn’t much you can do.
• Try to eliminate the reading of the page source as a potential contributor to the memory/performance problem by commenting it out.
• Use a streaming HTML parser such as Majestic 12– avoids the need to load the entire page source into memory (again, if you need to parse)!
• Limit the size of the pages you’re going to download, say, only download 150KB. The average page size is about 100KB-130KB

Additionally, can you tell us what’s your initial rate of fetching pages and what does it go down to? Are you seeing any errors/exceptions from the web request as you’re fetching pages?

Update

In the comment section I noticed that you’re creating thousands of threads and I would say that you don’t need to do that. Start with a small number of threads and keep increasing them until you peek the performance on your system. Once you start adding threads and the performance looks like it’s tapered off, then sop adding threads. I can’t imagine that you will need more than 128 threads (even that seems high). Create a fixed number of threads, e.g. 64, let each thread take a URL from your queue, fetch the page, process it and then go back to getting pages from the queue again.