Question

So I’m working with a file format. The file format contains chunks of data… So what I have is an array of List’s for the “chunks”. Those get added to when I add data to the class via a function.

Now, when I save the file, I need to ‘insert’ a chunk in the beginning. Now I know this may not make sense, but I need to add that chunk (which is blank) BEFORE I calculate the data offsets for the data types in the chunks. If I don’t, the data offsets get screwed up. After I insert that blank chunk, I then create a new byte[] array in which I copy the necessary data to, and then I “overwrite” the blank chunk that I inserted with the updated byte array.

The main reason why I need to do this is because the datachunk that I’m inserting contains offsets of the other data, so I need to create the offsets after everything has been added.

Basically what I have is this (only simplified):

    public struct SizeIndexPair {
        public int Size;
        public int Index;
    };

    public class Chunks {
        private Dictionary<int, SizeIndexPair> reserved;
        public List<List<byte> > DataChunks;

        ...

        public void Reserve(int ID, int size, int index) {
            SizeIndexPair sip;
            sip.Size = size;
            sip.Index = index;
            reserved.Add(ID, sip);
            List<byte> placeHolder = new List<byte>(size);
            DataChunks.Insert(index, placeHolder);
        }

        public void Register(int ID, byte[] data) {
            SizeIndexPair sip = reserved[ID];
            if (sip.Size != data.Length) 
                throw new IndexOutOfRangeException();
            for (int i = 0; i < data.Length; i++) {
                DataChunks[sip.Index][i] = data[i];
            }
        }
    };

(I’m using List(of byte) here because I might need to add extra data to an existing chunk)

I hope I’m making sense.

The problem with this approach is that I’m ‘doubling’ the array, which is eating up more memory. Plus the process of copying the data can slow down my app quite a bit, especially since the file typically contains alot of data.

Is there a better approach to this?

One thing that would have easily solved this is fixing the List and, instead of reserving/registering the array, I can simply access the array directly via a pointer. Is there a way to do this?

Thanks for your help.

Answer 1

If the goal is to insert a specific collection of bytes before the rest of the entries in your List<List<byte>> then you can use the overload:

DataChunks.Insert(0, prefix);

where 0 represents the index of the element in the collection to insert before and prefix is the value to insert.

Then to get the resulting byte stream:

foreach(byte b in DataChunks.SelectMany(c => c))
    Console.WriteLine(b); // replace with the method you use to write the `List of Lists of Bytes