Question

Originally i wanted to know whether ToList allocates more memory than using the constructor of List<T> which takes an IEnumerable<T> (no difference).

For test purposes i used Enumerable.Range to create a source array that i could use to create an instance of List<int> via 1.ToList and 2.constructor. Both are creating copies.

This is how I came to notice a great difference in memory consumption between:

1. Enumerable.Range(1, 10000000) or
2. Enumerable.Range(1, 10000000).ToArray()

When i use the first and call ToList the resulting object ineeds ~60% more memory than the Array(38,26MB/64MB).

Q: What is the reason for this or where is my error in reasoning?

var memoryBefore = GC.GetTotalMemory(true);
var range = Enumerable.Range(1, 10000000);
var rangeMem = GC.GetTotalMemory(true) - memoryBefore; // negligible
var list = range.ToList();
var memoryList = GC.GetTotalMemory(true) - memoryBefore - rangeMem;

String memInfoEnumerable = String.Format("Memory before: {0:N2} MB List: {1:N2} MB"
    , (memoryBefore / 1024f) / 1024f
    , (memoryList   / 1024f) / 1024f);
// "Memory before: 0,11 MB List: 64,00 MB"

memoryBefore = GC.GetTotalMemory(true);
var array = Enumerable.Range(1, 10000000).ToArray();
var memoryArray = GC.GetTotalMemory(true) - memoryBefore;
list = array.ToList();
memoryList = GC.GetTotalMemory(true) - memoryArray;

String memInfoArray = String.Format("Memory before: {0:N2} MB Array: {1:N2} MB List: {2:N2} MB"
   , (memoryBefore / 1024f) / 1024f
   , (memoryArray  / 1024f) / 1024f
   , (memoryList   / 1024f) / 1024f);
// "Memory before: 64,11 MB Array: 38,15 MB List: 38,26 MB"

Answer 1

This probably relates to the doubling algorithm used to resize the backing buffer when adding to a list. When you allocate as an array, the length of that is known, and can be queried by checking for IList[<T>] and/or ICollection[<T>]; thus it can allocate a single array, right-sized the first time, and then just block-copy the contents.

With the sequence this is not possible (the sequence does not expose the length in any accessible way); thus it must instead fall back to “keep filling up the buffer; if full, double it and copy”.

Obviously this needs approx double the memory.

An interesting test would be:

var list = new List<int>(10000000);
list.AddRange(Enumerable.Range(1, 10000000));

This will allocate the right size initially, while still using the sequence.

tl;dr; the constructor, when passed a sequence, first checks to see if it can obtain the length by casting to a well-known interface.