Home/ Questions/Q 7564155
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-30T13:50:43+00:00

At the company I work for, we are building a data warehousing application which

  • 0

At the company I work for, we are building a data warehousing application which will be a web based front end for a lot of queries that we run. [Using ASP.net]

Some of these queries will bring in over a million records (and by year end maybe around 2 million records) –

Most will be thousands, etc.

What is a good way to architect the application in such a way that you can browse to the query that you want, export it and have a CSV file generated of the data requested –

I was thinking a web based interface that calls BCP to generate a file, and shows you when the report has been created so that it can be downloaded, and expires within 24 hours of being created –

Any Ideas?

Sas

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    I architected something like this in a former life. Essentially a C# command-line app that ran queries against our reporting server.

    Some key points:

    • It didn’t matter how long they took because it was in the background – when a file was ready, it would show up in the UI and the user could download it. They weren’t waiting in real time.

    • It didn’t matter how inefficient the query was – both because of the point above, and because the reports were geared to the previous day. We ran them off of a reporting copy of production, not against production, which was kept on a delay via log shipping.

    • We didn’t set any expiration on the files because the user could request a report on a Friday and not expect to look at it until Monday. We’d rather have a file sitting around on the disk than run the report again (file server disk space is relatively cheap). We let them delete reports once they were done with them, and they would do so on their own to prevent clutter in the UI.

    • We used C# and a DataReader rather than bulk export methods because of various requirements for the data. We needed to provide the flexibility to include column headers or not, to quote data or not, to deal with thousand separators and decimal points supporting various cultures, to apply different line endings (\r\n, \n, \r), different extensions, Unicode / non-Unicode file formats, and different delimiters (comma, tab, pipe, etc). There were also different requirements for different customers and even for different reports – some wanted an e-mail notification as soon as a report was finished, others wanted a copy of the report sent to some FTP server, etc.

    There are probably a lot of requirements you haven’t thought of yet but I hope that gives you a start.

    • 0