Home/ Questions/Q 134297
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-11T06:35:16+00:00

I have a large database and want to implement a feature which would allow

  • 0

I have a large database and want to implement a feature which would allow a user to do a bulk update of information. The user downloads an excel file, makes the changes and the system accepts the excel file.

  1. The user uses a web interface (ASP.NET) to download the data from database to Excel.
  2. User modifies the Excel file. Only certain data is allowed to be modified as other map into the DB.
  3. Once the user is happy with their changes they upload the changed Excel file through the ASP.NET interface.
  4. Now it’s the server’s job to suck data from the Excel file (using Gembox) and validate the data against the database (this is where I’m having the trouble)
  5. Validation results are shown on another ASP.NET page after validation is complete. Validation is soft and so hard fails only occur when say an index mapping into DB is missing. (Missing data causes ignore, etc)
  6. User can decide whether the actions that will be taken are appropriate, in accepting these the system will apply the changes. (Add, Modify, or Ignore)

Before applying the changes and/or additions the user has made, the data must be validated to avoid mistakes by the user. (The accidentally deleted dates which they didn’t mean to)

It’s not far fetched for the rows that need updating to reach over 65k.

The question is: What is the best way to parse the data to do validation and to build up the change and addition sets?

If I load all data that the excel data must be validated against into memory I might unnecessarily be affecting the already memory hungry application. If I do a database hit for every tuple in the excel file I am looking at over 65k database hits.

Help?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. The approach I’ve seen used in the past is:

    1. Bulk-load the user’s data into a ‘scratch’ table in the database.
    2. Validate data in the scratch table via a single stored procedure (executing a series of queries), marking rows that fail validation, require update etc.
    3. Action the marked rows as appropriate.

    This works well for validating missing columns, valid key values etc. It’s not so good for checking the format of individual fields (don’t make SQL pull strings apart).

    As we know, some folk feel uncomfortable putting business logic in the database, but this approach does limit the number of database hits your application makes, and avoids holding all the data in memory at once.

    • 0