Question

I’ve decided that I need to create a 64-bit List to fulfill some of my program’s needs, namely the ability to use longs as indexes. I’ve looked at the Mono code for implementing a List, and have come to the general conclusion that no matter what I choose, I should make a variation of IList (using longs) to work with it.

Now my question is, what do you think would be a good approach to this design? I am currently thinking of two possibilities -> since a List is just a wrapper for the Array class, I can just rewrite the List class to use giant arrays; or I can write the class to use a List of Lists to maintain and grow the data as needed. The problem with the first appears to be choosing an array that is too large, and the second is trying to make Remove() and other assorted methods work, when I’d probably need to perform massive memory copies to keep everything indexed properly. Your thoughts?

Answer 1

Since arrays are limited to be indexed by int you will have to use multiples of them anyway. I’d directly go for List of List. Note that all objects in CLR are limited by 2Gb single chunk of memory for allocation.

Side notes:

• if you planning to implement true 4GB+ linear array that support insert/remove operations in the middle you simply should adjust your performace expectations – any insert or remove will be slow. At such scale having append only array maybe reasonable approach.
• if add/remove are important consider some other data structures i.e. B-tree. You will sacrify constant access time in most cases, but gain reasonable.
• if you arrays are sparse consider simple Dictionary instead or as backing storage.

Here is link to more detailed discussion about large arrays.