Question

I am running some experiments, timing them and comparing the times to find the best “algorithm”. The question that came up was if running the tasks in parallel would make the relative runningtimes of the experiments wrong and if I would get more representative results by running them sequentially. Here is a (simplified) version of the code:

public static void RunExperient(IEnumerable<Action> experiments)
    {
        Parallel.ForEach(experiments, experiment =>
        {
            var sw = Stopwatch.StartNew(); //line 1
            experiment();                  //line 2   
            sw.Stop();                     //line 3
            Console.WriteLine(@"Time was {0}", sw.ElapsedMilliseconds);
        });
    }

My questions are about what is happening “behind the scenes”:

1. When a task has started, is it possible that the OS or the framework can suspend the task during its execution and continue on later making the running time of the experiment all wrong?

2. Would I get more representative results by running the experiments sequentially?

Answer 1

That depends on the machine that you are running on and what the experiments do, but generally the answer is yes, they may affect one another. Mainly through resource starvation. Here’s an example:

public class Piggy { 
   public void GreedyExperiment() { 
       Thread.Priority = ThreadPriority.Highest;
       for (var i=0;i<1000000000;i++) {
           var j = Math.Sqrt(i / 5);
       }
   }
}

That’s going to do a tight loop on a high priority thread, which will basically consume one processor until it is done. If you only have one processor in the machine and TPL decides to schedule two experiments on it, the other one is going to be starved for CPU time.