alt text http://img185.imageshack.us/img185/2080/mydiagram.png
Maxima.exe is a Computer Algebra System built as a native code rather than a managed code.
MyService works as a socket server, it will instantiate a new process of Maxima for each browser submitting mathematics expression to Web Server.
I cannot use AppDomain here because Maxima is a native code. However I want security policies provided by AppDomain such as restriction to write data on file system.
My question is, how can I get the AppDomain-like security policies when I instantiate Maxima in a process rather than in an AppDomain?
Since Maxima.exe is a native executable, the only security policies you can apply are those supported by the base Windows operating system. Unfortunately, that means your options are pretty limited: the only really useful thing you can do is run Maxima.exe in the context of a user account with limited privileges. When it comes to disallowing write access to most of your system, that should be sufficient, though.
The dated-but-still-useful article Safe Impersonation With Whidbey shows how to run a worker function in the context of another user account: you would then start Maxima.exe from that worker function.
Running as a limited user will of course still allow various Maxima.exe instances to interfere with each other to a certain degree (i.e. overwrite files created in the instance working directory). Starting each instance with a unique, randomly-created working directory (e.g. based on a GUID) may offer sufficient security for your purposes: if not, you may actually need to create distinct user accounts for each instance (or at least have a pool of user accounts for that purpose).
In the end, it all comes down to "how would I solve my security problem using Windows base-only features", and unfortunately you won’t have access to the much nicer .NET security features or the facilities found on non-Windows platforms (jails, systrace, etc.).