I have a VB.Net program that reads in a flat file, and then parses line by line, formatting the data into different spreadsheets in an excel workbook (each line can be any 10+ different record types so I parse and put in appropriate excel sheet).
For smaller sized flat files (under 10mb), the parser works great. However, I am trying this on a file that is over 120mb (400k+ lines). While running, I will get an OleDBException saying that the spreadsheet is full. Now I am pretty confident that Excel can handle a much larger data set than a flat file. So I assume this exception is not giving me the true story as to what is really occuring.
I open a connection, and then parse each line in the file, inserting each row into the excel file. I assumed it would be bad performance wise to open/close the connection between each insert. Could this be causing the issue? Any ideas what I need to do to handle such a large file? There are cases where the flat file can be over 500mb.
To actually do the insert into excel, I am just doing the following (I construct an sql query based on the type of row and values parsed):
Dim conn As New OleDbConnection()
conn.ConnectionString = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + ExportLocation.Text + "\" + importFileName + "-PVF.xls;Extended Properties=""Excel 8.0;HDR=YES"""
conn.Open()
Dim cmd1 As New OleDbCommand()
cmd1.Connection = conn
…parser code…
cmd1.CommandText = "INSERT INTO " + rowType + " values (" + currentRowString + ")"
cmd1.ExecuteNonQuery()
Excel 8.0 is the same as Excel 97 which allows a maximum of 65,536 rows. You’re exceeding this.