Question

I’ve made a Service that reads in several databases and load it in to another Database

the memory usage is Huge so now I’m looking in to reducing the memory usage.

I have a piece of code where I read in two colums from two diferent tables in to a list
my question is there a more memory effecient way to do this.

Currently there are about 3.4 million rows in the list and this number will only rise from now since the database this is taken from will only become bigger.

the code I’m talking about is initialized like this and used in a foreach loop:

    private List<int[]> DataIDList_1;
    private List<int[]> DataIDList_2;
    private List<int[]> DataIDList_3;
    public DatabaseTransferService()
    {
        DataIDList_1= new List<int[]>();
        DataIDList_2= new List<int[]>();
        DataIDList_3= new List<int[]>();
    }
    public void injectValues(docID,statID )
    {
          DataIDList_1.Add(new int[] { docID, statID });
    }
    public void insertData(List<int[]> DataIDList)
    {
        foreach (int[] intArrData in DataIDList)
        {
         int DataID = intArrData[0];
//Inserting data in to database based on the list
        }
    }

I’ve edited out the code that is not essential for my question.
for clarification the list contain only ID’s on rows which I then use to fetch the row and insert the data from one db to another db.

Edit: I’ve left out the insert part of the database because it’s not the part I’m looking at right now. I know there are probably something to get there also but I am currently looking at the single largest average memory user which in this case is my list’s from the 3 databases.

Edit2: the idea I had was to load in all the ID’s of the 5 databases and then load them in to the statistic database while only getting one row based on ID’s in the list meaning the only thing I have in my memory is what I’m working on currently and ID’s of all the rows
and just to clarify if there is a change in any of the rows in the 5 databases, I’ll get to update the statistic database based on a revisionTable that I’ve also loaded the ID’s in on.

Thanks in advance.

Answer 1

Is this kind of an interface between two databases? Maybe you could make us of yield here:

public IEnumerable<MyRecord> GetFromOriginalDB()
{
  // some fetching magic
  yield return myRecord;
}

public void WriteToTargetDB(IEnumerable<MyRecord> records)
{
  foreach(var record in records)
  {
    // insert magic
  }
}

The records should not be kept in memory (except from the very current one). However, this does require a lot of refactoring, given it works at all…