Question

Using a Linux machine, how many files can one ‘mmap’ using python (what is the limit)?
Thanks.

Answer 1

There’s multiple limits in play. The first is the system-wide open files limit, /proc/sys/fs/nr_open. This is typically in the million-range.

This limit may continue to be counted against processes that open the file, set up the mapping, then close the file again.

The second is the per-process getrlimit(2) RLIMIT_NOFILE per-process number of open files. This limit is typically in the 1024-range.

This limit appears to be on the highest file descriptor that will be allocated to the process, so it probably won’t continue to be counted against processes that open the file, set up the mapping, then close the file again.

The third is the amount of address space available to the process (getrlimit(2) RLIMIT_AS and architecture-dependent (2, 3, or maybe 4 gigabytes (with the 4:4 address space patch) on 32-bit, or many many gigabytes on 64-bit). 3 gigabytes is typical on x86, and due to address space layout issues, you’ll have probably 2.5 gigs available for mmap.

Clever use of fork(2) could let you set up one process per mapped file, and have gigabytes upon gigabytes mapped simultaneously even on 32-bit processors, but the getrlimit(2) RLIMIT_NPROC number of processes per user limitation may limit you to a maximum number of open processes, and thus files.