Question

I am working with big multidimensional byte arrays (~500mb per array, like, an array with dimensions of [8,8192,8192]) and I’d like to read and write them into file for storage.

I tried using BinaryFormatter but is very slow (takes minutes to do).

I tried using BinaryWriter but it only takes in a single dimensional array. Now, in C, there was no problem passing multi-dimensional array as single-dimensional. In C#, from what I see, I have two options:

• Allocate another chunk of memory for single-dimensional array, copy data into it with for loops, then write this array into file using BinaryWriter
• Using for loops, write each individual byte into file using BinaryWriter

Obviously it would be much faster if i’d just use byte[] everywhere and instead of using myarray[i,j] use myarray[i+j*myarray_width] but that would require rewrite of whole class just for purpose of easier working of one set of I/O functions (Save/Load).

There’s gotta be a better way.

Answer 1

When it comes to fast serialization, unsafe code might come in handy. There are two techniques that can help here:

1. Do a memcpy from your byte[,] to a fresh byte[] that you can pass to FileStream.Write. This requires, of course, a temporary doubling of storage space and some copying. You could split this work into 64KB chunks, though.
2. PInvoke to the unmanaged WriteFile and pass it the FileStream.SafeHandle value. WriteFile takes an arbitrary pointer so you can directly write out your byte[,] (converted into a void*).

Option 2 is maximally fast (“zero-copy”).

Sidenote: Unsafe code comes in handy whenever you need to reinterpret bytes. This capability leads to some nice abstractions in C. Fortunately, C# has that capability, too.