Question

I’m working on a monte carol pricer and I need to improve the efficiency of the engine.

• MonteCarlo path are created by a third party library (in c++)
• Pricing is done in IronPython (script created by the end user)
• Everything else is driven by a c# application

the pricing process is as follow:

• C# application request the path and collect them
• C# application push the paths to the script, who price and return the values
• C# application display the result to the end user

The number and size of the paths collected are know in advance.

I have 2 solutions with some advantages and drawback:

1. Request path generation, for each path, ask the script to return the result and finaaly aggregate the results once all paths are processed
2. Request path generation, collect all of them, request the script to process all of them at once and retrun me the final price

The first solutions work fine in all scenarios but as the number of path requested increase the performance decrease (I think it’s due to the multiple call to ironpython)

The second solution is faster but can hit an “out of memory” exception (I think it’s not enough virtual memory addressing space) if the number of path requested is too large

I choose the middle ground and process a bunch of path then aggregate the prices.
What I want now is to increase the performance futher by knowing in advance how many path I can process withou hitting the “out of memory” exception

I did the math and I know in advance the size (in memory) of path for a given request. However because I’m quiet sure it’s not a memory problem but more virtual memory addressing issue

---

So all this text is summarize by the following 2 questions:

1. Is it possible to know in advance how much virtual memory address my
   process wil need to store n instance of a class (size in memory and structure are known)?
2. Is it possible to know how much virtual memory address are still available for my process

btw I’m working on the 32 bit computer

Thanks in advance for the help

Answer 1

Finding out how much memory an object takes in .NET is a pretty difficult task. I’ve hit the same problem several times. There are some imperfect methods, but none are very precise.

My suggestion is to get some estimate of how much a path will take, and then pass a bunch of them leaving a good margin of safety. Even if you’re processing them just 10 at a time, you’ve reduced the overhead 10 times already.

You can even make the margin configurable and then tweak it until you strike a good balance. An even more elegant solution would be to run the whole thing in another process and if it hits an OutOfMemoryException, restart the calculation with less items (and adjust the margin accordingly). However, if you have so much data that it runs out of memory, then it might be a bit slow to pass it across two processes (which will also duplicate the data).

Could it be that the memory overflow is because of some imperfections in the path processor? Memory leaks maybe? Those are possible both in C++ and .NET.