Question

Currently I am debugging a program in linux that looks like this:

int main(){
    loadHugeFile();
    processTheDataOfTheFile();
    return 0;
}

The thing is loadHugeFile function needs to load a very huge file in gigabytes that takes about 5 minutes, while processTheDataOfTheFile takes less than 10 seconds to calculate the required data and return some values. In the future, the file’s size might increase even further, and it will take even more time to load. The file is an invert index, so the whole file is needed.

Is it possible to have one process load this file into the RAM, retain it and have any other process access this part of the loaded file? This is to skip that many minutes loading. I recall Windows has this function that allows you to access/modify another process’s memory, but what are my available choices here in linux?

Answer 1

You can use mmap function.

In computing, mmap(2) is a POSIX-compliant Unix system call that maps files or devices into memory. It is a method of memory-mapped file I/O.

You got 2 advantages. Extreme speed in loading file and the content will be in a memory area that can be shared between many other processes (just use mmap with the flag MAP_SHARED).

You can test the speed of mmap with this short and dirty code. Just compile it and exec it passing the file you want to load as paramenter.

#include <stdio.h>
#include <stdint.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/mman.h>

int main(int argc, char *argv[])
{
    struct stat sb;

    int fd = open(argv[1], O_RDONLY);

    // get the size in bytes of the file
    fstat (fd, &sb);

    // map the file in a memory area
    char *p = mmap (0, sb.st_size, PROT_READ, MAP_SHARED, fd, 0);

    // print 3 char of the file to demostrate it is loaded ;)
    printf("first 3 chars of the file: %c %c %c\n", p[0], p[1], p[2]);

    close(fd);

    // detach
    munmap(p, sb.st_size);
}