Question

I need to insert 800000 records into an MS Access table. I am using Delphi 2007 and the TAdoXxxx components. The table contains some integer fields, one float field and one text field with only one character. There is a primary key on one of the integer fields (which is not autoinc) and two indexes on another integer and the float field.

Inserting the data using AdoTable.AppendRecord(...) takes > 10 Minutes which is not acceptable since this is done every time the user starts using a new database with the program. I cannot prefill the table because the data comes from another database (which is not accessible through ADO).

I managed to get down to around 1 minute by writing the records to a tab separated text file and using a tAdoCommand object to execute

insert into table (...) select * from [filename.txt] in "c:\somedir" "Text;HDR=Yes"

But I don’t like the overhead of this.

There must be a better way, I think.

EDIT:

Some additional information:

• MS Access was chosen because it does not need any additional installation on the target machine(s) and the whole database is contained in one file which can be easily copied.
• This is a single user application.
• The data will be inserted only once and will not change for the lifetime of the database. Though, the table contains one additional field that is used as a flag to indicate that the corresponding record in another database has been processed by the user.
• One minute is acceptable (up to 3 minutes would be too) and my solution works, but it seems too complicated to me, so I thought there should be an easier way to do this.
• Once the data has been inserted, the performance of the table is quite good.
• When I started planning/implementing the feature of the program working with the Access database the table was not required. It only became necessary later on, when another feature was requested by the customer. (Isn’t that always the case?)

EDIT:

From all the answers I got so far, it seems that I already got the fastest method for inserting that much data into an Access table. Thanks to everybody, I appreciate your help.

Answer 1

Your text based solution seems the fastest, but you can get it quicker if you could get an preallocated MS Access in a size near the end one. You can do that by filling an typical user database, closing the application (so the buffers are flushed) and doing a manual deletion of all records of that big table – but not shrinking/compacting it.

So, use that file to start the real filling – Access will not request any (or very few) additional disk space. Don’t remeber if MS Access have a way to automate this, but it can help much…