Question

Here’s an interesting problem. I have an ETL script written in c# that I have been running manually on a somewhat regular basis. It is used to update my web app’s database.

I want to automate the ETL process AND create an interface for the web app admins to manually start the ETL process.

I could have sql server kick off the ETL process on a schedule and implement a button or something on a web page that will do the same thing, but I don’t want to put my code in 2 different places because I don’t want to update it in 2 places when it changes. But I don’t know how to make my web app tell SQL server to manually start a scheduled process. Can this be done?

OR

I could somehow implement the scheduling in the web app itself, but by now most people are familiar with the problems that are faced when trying that (app may not be running at certain times, must wait on request to start a process (without some trickery)). Also, since the ETL process takes a while, I don’t want to make some poor end user wait on a response, so it would definitely have to use a new thread.

What else could I do? What would you do?

Answer 1

You mentioned you are using SQL Server. What version is it? Is SQL Server Integration Services (which is the full-blown ETL tool for the platform) an option for you? If you want scheduled Extraction, Transformation and Load jobs, SSIS is a great tool on the Microsoft platform and is included in most SQL Server licenses.

Or, check out Quartz.