Home/ Questions/Q 8380301
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-09T16:25:20+00:00

I have a program producing a lot of data, which it writes to a

  • 0

I have a program producing a lot of data, which it writes to a csv file line by line (as the data is created). If I were able to open the csv file in excel it would be about 1 billion cells (75,000*14,600). I get the System.OutOfMemoryException thrown every time I try and access it (or even create an array this size). If anyone has any idea how to can take the data into vb.net so I can do some simple operations (all data needs to be available at once) then I’ll try every idea you have.

I’ve looked at increasing the amount of ram used but other articles/posts say this will run short way before the 1 billion mark. There’s no issues with time here, assuming it’s no more than a few days/weeks I can deal with it (I’ll only be running it once or twice a year). If you don’t know anyway to do it the only other solutions I can think of would be increasing the number of columns in excel to ~75,000 (if that’s possible – can’t write the data the other way around), or I suppose if there’s another language that could handle this?

At present it fails right at the start:

 Dim bigmatrix(75000, 14600) As Double

Many thanks,
Fraser 🙂

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    First, this will always require a 64bit operating system and a fairly large amount of RAM, as you’re trying to allocate about 8 GB.

    This is theoretically possible in Visual Basic targeting .NET 4.5 if you turn on gcAllowVeryLargeObjects. That being said, I would recommend using a jagged array instead of a multidimensional array if possible, as this will remove the requirement of needing a single allocation of 8GB. (This will also potentially allow it to work in .NET 4 or earlier.)

    • 0