Question

I run simulations on a Windows 64bit-computer with 64 GB RAM. Memory use reaches 55% and after a finished simulation run I remove all objects in the working space by rm(list=ls()), followed by a gc().

I supposed that this would free enough memory for the next simulation run, but actually memory usage drops by just 1%. Consulting a lot of different fora I could not find a satisfactory explanation, only vague comments such as:

"Depending on your operating system, the freed up memory might not be returned to the operating system, but kept in the process space."

I’d like to find information on:

1. which OS and under which conditions freed memory is not returned to the OS, and
2. if there is any other remedy than closing R and start it again for the next simulation run?

Answer 1

How do you check memory usage? Normally virtual machine allocates some chunk of memory that it uses to store its data. Some of the allocated may be unused and marked as free. What GC does is discovering data that is not referenced from anywhere else and marking corresponding chunks of memory as unused, this does not mean that this memory is released to the OS. Still from the VM perspective there’s now more free memory that can be used for further computation.

As others asked did you experience out of memory errors? If not then there’s nothing to worry about.

EDIT:
This and this should be enough to understand how memory allocation and garbage collection works in R.

From the first document:

Occasionally an attempt is made to release unused pages back to the
operating system. When pages are released, a number of free nodes
equal to R_MaxKeepFrac times the number of allocated nodes for each
class is retained. Pages not needed to meet this requirement are
released. An attempt to release pages is made every R_PageReleaseFreq level 1
or level 2 collections.

EDIT2:

To see used memory try running gc() with verbose set to TRUE:

gc(verbose=T)

Here’s a result with an array of 10’000’000 integers in memory:

Garbage collection 9 = 1+0+8 (level 2) ... 
10.7 Mbytes of cons cells used (49%)
40.6 Mbytes of vectors used (72%)
          used (Mb) gc trigger (Mb) max used (Mb)
Ncells  198838 10.7     407500 21.8   350000 18.7
Vcells 5311050 40.6    7421749 56.7  5311504 40.6

And here’s after discarding reference to it:

Garbage collection 10 = 1+0+9 (level 2) ... 
10.7 Mbytes of cons cells used (49%)
2.4 Mbytes of vectors used (5%)
         used (Mb) gc trigger (Mb) max used (Mb)
Ncells 198821 10.7     407500 21.8   350000 18.7
Vcells 310987  2.4    5937399 45.3  5311504 40.6

As you can see memory used by Vcells fell from 40.6Mb to 2.4Mb.