Question

I have a CSV dump from another DB that looks like this (id, name, notes):

1001,John Smith,15 Main Street
1002,Jane Smith,”2010 Rockliffe Dr.
Pleasantville, IL
USA”
1003,Bill Karr,2820 West Ave.

The last field may contain carriage returns and commas, in which case it is surrounded by double quotes. And I need to preserve those returns and commas.

I use this code to import CSV into my table:

BULK INSERT CSVTest
FROM 'c:\csvfile.csv'
WITH
(
  FIELDTERMINATOR = ',',
  ROWTERMINATOR = '\n'
)

SQL Server 2005 bulk insert cannot figure out that carriage returns inside quotes are not row terminators.
How to overcome?

---

UPDATE:
Looks like the only way to keep line breaks inside a field is to use different row separator. So, I want to mark all row separating line breaks by putting a pipe in front of them. How can I change my CSV to look like this?

1001,John Smith,15 Main Street|
1002,Jane Smith,”2010 Rockliffe Dr.
Pleasantville, IL
USA”|
1003,Bill Karr,2820 West Ave.|

Answer 1

OK, here’s a small Java program that I end up writing to solve the problem.
Comments, corrections and optimizations are welcome.

import java.io.*;

public class PreBulkInsert
{
    public static void main(String[] args)
    {
        if (args.length < 3)
        {
            System.out.println ("Usage:");
            System.out.println ("  java PreBulkInsert input_file output_file separator_character");
            System.exit(0);
        }

        try
        {
            boolean firstQuoteFound = false;
            int fromIndex;
            int lineCounter = 0;
            String str;

            BufferedReader in = new BufferedReader(new FileReader(args[0]));
            BufferedWriter out = new BufferedWriter(new FileWriter(args[1])); 
            String newRowSeparator = args[2];

            while ((str = in.readLine()) != null)
            {
                fromIndex = -1;
                do
                {
                    fromIndex = str.indexOf('"', fromIndex + 1);
                    if (fromIndex > -1)
                        firstQuoteFound = !firstQuoteFound;
                } while (fromIndex > -1);

                if (!firstQuoteFound)
                    out.write(str + newRowSeparator + "\r\n");
                else
                    out.write(str + "\r\n");
                lineCounter++;
            }
            out.close();
            in.close();
            System.out.println("Done! Total of " + lineCounter + " lines were processed.");
        }
        catch (IOException e)
        {
            System.out.println(e.getMessage());
            System.exit(1);
        }       
    }
}