Question

Environment : .net 4.0

I have a task that transforms XML files with a XSLT stylesheet, here is my code

public string TransformFileIntoTempFile(string xsltPath, 
    string xmlPath)
{
    var transform = new MvpXslTransform();
    transform.Load(xsltPath, new XsltSettings(true, false), 
        new XmlUrlResolver());

    string tempPath = Path.GetTempFileName();

    using (var writer = new StreamWriter(tempPath))
    {
        using (XmlReader reader = XmlReader.Create(xmlPath))
        {
            transform.Transform(new XmlInput(reader), null, 
                new XmlOutput(writer));
        }       
    }

    return tempPath;
}

I have X threads that can launch this task in parallel.
Sometimes my input file are about 300 MB, sometimes it’s only a few MB.

My problem : I get OutOfMemoryException when my program try to transform some big XML files in the same time.

How can I avoid these OutOfMemoryEception ? My idea is to stop a thread before executing the task until there is enough available memory, but I don’t know how to do that. Or there is some other solution (like putting my task in a distinct application).

Thanks

Answer 1

I don’t recommend blocking a thread. In worst case, you’ll just end up starving the task that could potentially free the memory you needed, leading to deadlock or very bad performance in general.

Instead, I suggest you keep a work queue with priorities. Get the tasks from the Queue scheduled fairly across a thread pool. Make sure no thread ever blocks on a wait operation, instead repost the task to the queue (with a lower priority).

So what you’d do (e.g. on receiving an OutOfMemory exception), is post the same job/task onto the queue and terminate the current task, freeing up the thread for another task.

A simplistic approach is to use LIFO which ensures that a task posted to the queue will have ‘lower priority’ than any other jobs already on that queue.