Question

I need to store quite large amount of binary data in database (MS SQL), and interact with this database via EF. Unfortunately, EF doesn’t support FILESTREAM (more exactly, there’s no streaming support).

So, I’ve decided to store data in chunks. Chunk is just an entity type:

public class Data
{
  public int Id { get; set; }
  public Collection<Chunk> Chunks { get; set; }
}

public class Chunk
{
  public int Id { get; set; }
  public int DataId { get; set; }
  public byte[] Content { get; set; }
}

First, I wanted to limit chunk size by some optimal value, let’s say, 1 Mb.
But then I’ve remembered about large objects and LOH.

As far as I understand, every Chunk.Content instance will be considered as large object with following consequences (memory fragmentation, particularly). Hence, intensive creation of Chunk objects finally will cause OutOfMemoryException (it is a “24/7” application, and work with that binary data is the main purpose of application).

I can’t reuse any buffer for data chunks, because I’m using EF…
Should I decrease chunk size, making it lower, then 85K?
Or is it a paranoia? 🙂

Answer 1

LOH fragmentation issue can occur only if you have too many chunks in the memory in the same time but from your description it looks like you will have one chunk per “process”. How many concurrent processes do you expect to have? If you expect to have many processes you most probably also expect HW with enough processing power and memory (64bit platform). Isn’t the HW a real limitation to have so many parallel chunks processed?

Anyway if you need to store stream to database you should simply use FILESTREAM without EF. Staying with EF at all costs is architecture failure.