Question

I want to use a library that uses file descriptors as the basic means to access its data. For performance reasons, I don’t want to have to commit files to the disk each before I use this library’s functions.

I want to create (large) data blobs on the fly, and call into the library to send them to a server. As it stands, I have to write the file to disk, open it, pass the FD to the library, wait for it to finish, then delete the file on disk. Since I can re-create the blobs on demand (and they’re not so large that they cause excessive virtual memory paging), saving them to disk buys me nothing, and incurs a large performance penalty.

Is it possible to assign a FD to a block of data that resides only as a memory-mapped entity?

Answer 1

You could mount a memory-backed filesystem: http://lists.apple.com/archives/darwin-kernel/2004/Sep/msg00004.html

Using this mechanism will increase memory pressure on the system, and will probably be paged out if memory pressure is great enough. It might be worthwhile to make it a configuration option, in case the user would rather some other application have first-choice of the memory.

Another option is to use POSIX shared memory segments: http://opengroup.org/onlinepubs/007908799/xsh/shm_open.html (I haven’t used POSIX shared memory segments myself; if I understand them correctly, they were designed to solve exactly this problem.)

The shm_open() function creates a memory object and returns a file descriptor. You could then mmap(2) that file descriptor, do your work, and pass the file descriptor to the library.

Don’t forget to shm_unlink the object when you’re done; POSIX shared memory segments, message queues, and semaphore arrays don’t automatically go away when the last process exits.