Question

I am working on a c parser and wondering how expert manage large amount of text / string (> 100mb) to store in memory?
the content is expected to be accessible all the time in fast pace.
bg: redhat / gcc / libc

a single char array would be out of boundary causing segmentation fault…
any idea or experience is welcomed to share / discuss…

Answer 1

mmap is the best way to deal with a large amount of data that is stored in a file, if you want random access to that data.

mmap tells the virtual memory system to map a contiguous portion of address space to contain the data found in the file. The virtual memory system will allocate a range of address space, backed by that file. When you access any location in that address space, it will allocate a page of physical memory, read that section of the file in from the disk, and point that portion of your virtual address space to the physical memory that it used to read the file. When it needs to make more room in physical memory, it will write out any changes to disk (if applicable), and remove the mapping of that section of virtual address space.

You would use it like so:

#include <sys/types.h>
#include <sys/stat.h>
#include <sys/mman.h> /* the header where mmap is defined */
#include <fcntl.h>

int file;
char *contents;
struct stat statbuf;
off_t len;

file = open("path/to/file", O_RDONLY);
if (file < 0)
  exit(1); /* or otherwise handle the error */

if (fstat(file, &statbuf) < 0)
  exit(1);

len = statbuf.st_size;

contents = mmap(0, len, PROT_READ, MAP_SHARED, file, 0);
if (contents == MAP_FAILED)
  exit(1);

// Now you can use contents as a pointer to the contents of the file

// When you're done, unmap and close the file.

munmap(contents, len);
close(file);