Question

I have a DataTable thats filling up with 360,000 rows of SQL Data (this is intended). However this runs into OOM issues.

This is what I have, however, i’m not sure on how to handle everthing after the last interval of 1000. Or maybe there is a better way

int catchInt = 0;
string combineWhereClause = string.Empty;

for (int i = 0; i < ThousandLoopTable.Rows.Count; i++)
{
    catchInt++;
    combineWhereClause = combineWhereClause + 
                        "','" + 
                         ThousandLoopTable.Rows[i].ItemArray[0].ToString();

    if (catchInt >= 1000)
    {
        catchInt = 0;
        combineWhereClause = combineWhereClause.TrimStart('\'');
        combineWhereClause = combineWhereClause.TrimStart(',');

        Directory.CreateDirectory(ExportDirectory);
        SQLProcessing.SQLProcessor.MasterSqlConnection = 
            SQLProcessing.SQLProcessor.OpenMasterSqlConnection(SQLServer);
        DataTable dtTable = 
            SQLProcessing.SQLProcessor.QueryDataTable(sql_selectionquery);
        for (int m = 0; m < dtTable.Rows.Count; m++)
        {
            string FileName = dtTable.Rows[m].ItemArray[0].ToString() + ".txt";
            string OCR = dtTable.Rows[m].ItemArray[1].ToString();
            File.AppendAllText(ExportDirectory + "\\" + FileName, OCR);

        }

        combineWhereClause = string.Empty;
    }
}

So for example if there is 3120 rows, this will do 3000, but will not do the last 120. However, i’m not sure how to handle the last 120 because I don’t really want to do that in the for loop do I?

Answer 1

There are probably better ways to handle data (see other answers), but I think this is what you need for your current approach:

Don’t reset catchInt for every batch. Instead, initialise it as 1 and let it run as a counter for the whole operation. Then change the if to:

if (catchInt % 1000 == 0 || catchInt == ThousandLoopTable.Rows.Count)
{
    // Execute your batch
}

This uses the Modulus operator to identify when catchInt is divisible by 1000.