Question

My problem is simple in nature- query a database, and write output to a set of files (~1000).

I have a SQL query I need to execute from C# on SQL Server. The query is fairly simple except for one of the values in an IN clause.

As an example, the query may look like:

SELECT 
    ID, Name
FROM
    MyTable
WHERE
    SomeValue IN (/* list of values */)

The list of values is around 300 strings that are 6 characters each. The values will ultimately look something like (“A00001”, “A00002”, etc), and are specified as input by the user of the application.

My initial thought was to build the query out as a long string, with each of the values aggregated together. This doesn’t seem like the proper approach however, and seems to be error prone. My next thought was to use a stored procedure, but I’m unsure of how the data should be passed into it. How should this query be built in an efficient way where the data can be reliably passed in?

My other option is to read in ALL data from SQL Server and then filter the results in C# as I receive them. If I do this, however, I’ll get around 9 million records which seems to be too many to read into memory as a data set. Because I’m writing files after parsing ~5000 records, I don’t want to keep a data reader connected to the database while I chew through the data I’m receiving.

What is the best approach?

Answer 1

Here is one way to do it using xml:

C#:

StringBuilder sb = new StringBuilder();
sb.AppendLine("<SelectedValues>");

foreach (string s in userSelectedValues)
  sb.AppendFormat("<row val=\"{0}\" />{1}", s, Environment.NewLine);
sb.AppendLine("</SelectedValues>");

Pass this sb.ToString() AS XML TO your stored procedure

SQL:

CREATE PROCEDURE dbo.MyStoredProcedure
(
  @UserSelectedValuesXml XML
)     

SELECT * 
FROM dbo.MyTable
WHERE Id IN ( SELECT ref.value('@val', 'varchar')
              FROM @UserSelectedValuesXml.nodes('/SelectedValues/row') AS T(ref))