Question

So, I have a huge query that I need to run on an Access DB. I am attempting to use a for loop to break it down because I can’t run the query all at once (it has an IN with 50k values). The reader is causing all kinds of problems hanging and such. Most times when I break up the for loop into 50-10000 values the reader will read 400 (exactly 400) values and then hang for about 3 minutes then do another hundred or so, hang, ad infinium. If I do over 10k values per query it gets to 2696 and then hangs, does another 1k or so after hanging, on and on. I have never really worked with odbc, sql or any type of database for that matter, so it must be something stupid, or is this expected? Maybe there’s a better way to do something like this? Here’s my code that is looped:

//connect to mdb
OdbcConnection mdbConn = new OdbcConnection();
mdbConn.ConnectionString = @"Driver={Microsoft Access Driver (*.mdb)};DBQ=C:\PINAL_IMAGES.mdb;";
mdbConn.Open();

OdbcCommand mdbCmd = mdbConn.CreateCommand();
mdbCmd.CommandText = @"SELECT RAW_NAME,B FROM 026_006_886 WHERE (B='CM1' OR B='CM2') AND MERGEDNAME IN" + imageChunk;
OdbcDataReader mdbReader = mdbCmd.ExecuteReader();

while (mdbReader.Read())
{
  sw.WriteLine(@"for /R %%j in (" + mdbReader[0] + @") do move %%~nj.tif .\" + mdbReader[1] + @"\done");
  linesRead++;
  Console.WriteLine(linesRead);
}
mdbConn.Close();

Here’s how I populate the imageChunk variable for the IN by reading 5000 lines with a value line from a text file using a StreamReader:

                string imageChunk = "(";
                for (int j = 0; j < 5000; j++)
                {
                    string image;
                    if ((image = sr.ReadLine()) != null)
                    {
                        imageChunk += @"'" + sr.ReadLine() + @"',";
                    }
                    else
                    {
                        break;
                    }
                }
                imageChunk = imageChunk.Substring(0, imageChunk.Length - 1);
                imageChunk += ")";

Answer 1

Ended up using a data adapter… Was slow but provided constant feedback instead of freezing up. Never really got a good answer why, but got some advice on smarter ways to perform a large query.