I have a web page that accepts an excel file, reads data from it and stores it in a buffer table. Then a procedure is called that reads from this buffer table and each record passes through number of validations. After procedure completes, I delete the buffer table contents from code behind. I have executed this code using about 100 records and it executes in a few seconds. However, when the record size is increased (say about 2000), the procedure takes over 5 mins to execute, but the web page hangs. I have checked the table, the record insertion and buffer table deletion takes about 6-7 mins, but the web page does not return a result even after 30 mins. I have tried to optimize the procedure, but still in case of large number of records the web page hangs.
Please give me some direction as to how to avoid this page hanging situation. Any help would be great. Thanks in advance
I think that the first thing you should do is wrap your inserts into a transaction.
If there are too many records for a single transaction, you could perform a commit every n records (say 500).
As far as the web page returning, you could be reaching a timeout of some sort where IIS or the client abandons the request or if you update the page with data, you could have invalid data which is causing errors in the page.
For this, you should check the windows event log to see if IIS or ASP.Net are reporting any exceptions. You can also run fiddler to see what is happening to the request.
Finally, I would strongly suggest a redesign that does not require the user to wait with a submitted form on the screen until processing is complete.
The standard pattern that we use for this type of functionality is to record incoming requests in the database with a GUID, kick off a background worker to perform the task, and return the GUID to the client.
When the background worker has finished (or encounters an error), it updates the request table in the database with the new status (i.e. success or fail) and the error message if any.
The client can use the GUID to issue ajax requests to the web server on a regular basis (using window.timeout so as not to block the user and allow animations to be displayed) to determine whether or not the process is complete. Once the process is complete, the UI can be updated as needed.
Update
To record incoming requests in the database with a GUID, create a table that contains a GUID column as the primary key, a status column (3 values: in progress, success, failure), and an error message column.
When the request is received, create a new GUID in your code, then write a record to this new table with this GUID and a status of in progress prior to launching the background worker.
You will pass the GUID to the background worker so that it can update the table on completion (it just updates that status to complete or error and records the error message, if any).
You will also pass the GUID back to the client through javascript so that the client can periodic ask the web server to perform a query against the table using the GUID to determine when the request is no longer in progress.