We have a process that is getting data in real time and adding records to a database. We’re using SQL Server 2008 Integration Services to run our Extract Transform Load (ETL) process. We download about 50 files from an FTP site, process them and then archive the files.
The problem is that the processing is taking about 17s per file even thought the files are really small (about 10 lines) and the processing code is fairly simple. Looking at the load on the machine it is CPU bound and there are not a lot of traffic on the network, disc, or memory.
I suspect that SSIS might be re-compiling the C# code every time it is run. Has anyone run into similar problems? Or have you used a similar process without problems?
Are there any tools that can allow us to profile a dtsx package?
Since you’re using SSIS 2008, your Script Tasks are always precompiled.