Question

I want to import my IIS logs into SQL for reporting using Bulk Insert, but the comment lines – the ones that start with a # – cause a problem becasue those lines do not have the same number f fields as the data lines.

If I manually deleted the comments, I can perform a bulk insert.

Is there a way to perform a bulk insert while excluding lines based on a match such as : any line that beings with a “#”.

Thanks.

Answer 1

The approach I generally use with BULK INSERT and irregular data is to push the incoming data into a temporary staging table with a single VARCHAR(MAX) column.

Once it’s in there, I can use more flexible decision-making tools like SQL queries and string functions to decide which rows I want to select out of the staging table and bring into my main tables. This is also helpful because BULK INSERT can be maddeningly cryptic about the why and how of why it fails on a specific file.

The only other option I can think of is using pre-upload scripting to trim comments and other lines that don’t fit your tabular criteria before you do your bulk insert.