Question

I need to serialize a function in Erlang, send it over to another note, deserialize and execute it there. The problem I am having is with files. If the function reads from a file which is not in the second node I am getting an error. Is there a way how I can differentiate between serializable and not serializable constructs in Erlang? Thus if a function makes use of a file or pid, then it fails to serialize?

Thanks

Answer 1

First of all, if you are sending anonymous functions, be extremely careful with that. Or, rather, just don’t do it

There are a couple of cases when this function won’t even be executed or will be executed in a completely wrong way.

Every function in Erlang, even an anonymous one, belongs to some module, in the one it’s been constructed inside, to be precise. If this function has been built in REPL, it’s bound to erl_eval module, which is even more dangerous (I’ll explain further, why).

Say, you start two nodes, one of them has a module named ‘foo’, and one doesn’t have such a module loaded (and cannot load it). If you construct a lambda inside the module ‘foo’, send it to the second node and try to call it, you’ll fail with {error, undef}.

There can be another funny problem. Try to make two versions of module ‘foo’, implement a ‘bar’ function inside each of them and implement a lambda inside of which (but the lambdas will be different). You’ll get yet another error when trying to call a sent lambda.

I think, there could possibly be other tricky parts of sending lambdas to different nodes, but trust me, that’s already quite a lot.

Secondly, there are tons of way you can get a process or a port inside a lambda without knowing it in advance

Even though there is a way of catching closured variables from a lambda (if you take a look at a binarized lambda, all the external variables that are being used inside it are listed starting from the 2nd byte), they are not the only source of potential pids or ports.

Consider an easy example: you call a self() function inside your lambda. What will it return? Right, a pid. Okay, we can probably parse the binary and catch this function call, along with a dozen of other built-in functions. But what will you do when you are calling some external function? ets:lookup(sometable, somekey)? some_module:some_function_that_returns_god_knows_what()? You don’t know what they are going to return.

Now, to what you can actually do here

1. When working with files, always send filenames, not descriptors. If you need file’s position or something, send it as well. File descriptors shouldn’t be known outside the process they’ve been opened.

2. As I mentioned, do everything to avoid lambdas to be sent to other nodes. It’s hard to tell how to avoid that, since I don’t know your exact task. Maybe, you can send a list of functions to execute, like:

   [{module1, parse_query},
    {module1, dispatch_parsed_query},
    {module2, validate_response},
    {module2, serialize_query}]
   

   and pass arguments through this functions sequence (be sure all the modules exist everywhere). Maybe, you can actually stick to some module that is going to be frequently changed and deployed over the entire cluster. Maybe, you might want to switch to JS/Lua and use externally started ports (Riak is using spidermonkey to process JS-written lambdas for Map/Reduce requests). Finally, you can actually get module’s object code, send it over to another node and load it there. Just keep in mind it’s not safe too. You can break some running processes, lose some constructed lambdas, and so on.