Question

I have a tab delimited txt file with 500K records. I’m using the code below to read data to dataset. With 50K it works fine but 500K it gives “Exception of type ‘System.OutOfMemoryException’ was thrown.”

What is the more efficient way to read large tab delimited data?
Or how to resolve this issue? Please give me an example

public DataSet DataToDataSet(string fullpath, string file)
{
    string sql = "SELECT * FROM " + file; // Read all the data
    OleDbConnection connection = new OleDbConnection // Connection
                  ("Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + fullpath + ";"
                   + "Extended Properties=\"text;HDR=YES;FMT=Delimited\"");
    OleDbDataAdapter ole = new OleDbDataAdapter(sql, connection); // Load the data into the adapter
    DataSet dataset = new DataSet(); // To hold the data
    ole.Fill(dataset); // Fill the dataset with the data from the adapter
    connection.Close(); // Close the connection
    connection.Dispose(); // Dispose of the connection
    ole.Dispose(); // Get rid of the adapter
    return dataset;
}

Answer 1

Use a stream approach with TextFieldParser – this way you will not load the whole file into memory in one go.