Question

I am trying to process 114,000 rows in a dataset (populated from an oracle database). I am hitting an error at around the 600 mark – “Thread was being aborted”.
All I am doing is reading the dataset, and I still hit the issue. Is this too much data for a dataset? It seems to load into the dataset ok though. I welcome any better ways to process this amount of data.

rootTermsTable = entKw.GetRootKeywordsByCategory(catID);
for (int k = 0; k < rootTermsTable.Rows.Count; k++)
{
    string keywordID = rootTermsTable.Rows[k]["IK_DBKEY"].ToString();
    ...
}


public DataTable GetKeywordsByCategory(string categoryID)
    {
        DbProviderFactory provider = DbProviderFactories.GetFactory(connectionProvider);
        DbConnection con = provider.CreateConnection();
        con.ConnectionString = connectionString;

        DbCommand com = provider.CreateCommand();
        com.Connection = con;
        com.CommandText = string.Format("Select * From icm_keyword WHERE (IK_IC_DBKEY = {0})",categoryID);
        com.CommandType = CommandType.Text;

        DataSet ds = new DataSet();
        DbDataAdapter ad = provider.CreateDataAdapter();
        ad.SelectCommand = com;

        con.Open();
        ad.Fill(ds);
        con.Close();

        DataTable dt = new DataTable();
        dt = ds.Tables[0];

        return dt;

        //return ds.Tables[0].DefaultView;

    }

Answer 1

A few thoughts:

1. Depending on what you are doing, a DataReader may be faster than a DataTable. A DataReader allows you to process one row at a time as it is read from the database. If you are doing a fair bit of processing on each row, then your current method (a DataTable) is likely the better approach, as it allows you to pull all rows into memory and process them without keeping the connection open.
2. Consider “SELECT IK_DBKEY” instead of “SELECT *”, as you will be pulling back less data over the wire and into memory.
3. Finally, you might put a try/catch around your code to ensure it is the actual source of the error message. If you are using ASP.Net , “Thread was being aborted” is a pretty common error, which usually indicates that a processing request was interrupted (e.g. someone navigated away from away from a long-running request).