Question

I have a program set up like this. It is a .Net Framework 4 console app.
This program is used to gather the sc-bytes and cs-bytes from every log file (from the previous week) on each server. This program is finished, but takes a very long time to run.

foreach (string server in servers)
            {

                foreach (string website in Directory.GetDirectories(server))
                {

                    foreach (string file in Directory.GetFiles(website))
                    {

I was just wondering if something like threading or PLINQ could be used to speed up the process?
If this would improve performance, I’m not sure of the best way to implement it because it seems illogical to have a new thread for each log file (or even each website) because having that many threads would obviously not improve performance.

If you need to see more code, please ask, but this program basically reads each line of each file that was created within the last seven days, adds the bytes numbers, and then uses directory entries to get the website name (from the ID), and then outputs the name and total bytes for each website into a text file (which will eventually be a database instead).
I don’t need any actual code, just advice on the best way (if possible) to improve performance.

Thanks.

Answer 1

In an IO bound task like this (iterating over directories and files and reading them), the bottleneck is disk IO, not CPU.

Parallelizing (?) this is not likely to help with speeding it up and might even hurt performance.