Question

I’m writing code to import dozens of moderate sized text files into SQL SERVER on a daily basis. Currently, these are imported into FOXPRO databases. I’m converting to use SQL SERVER. I have completed all the fixed length files, but the last three data files have a variable length field as the last field in each row. This final field can be up to length 32,000 bytes. In the database I have declared this as VARCHAR(MAX).

The field terminator is the row terminator which is a line feed. That is, the line terminates with a single ‘0x0a’ to mark both the end of the field and the end of the line, not two linefeeds.

Here’s the SQL I’m using:

BULK INSERT 
  [MyDB].[dbo].[X]
  FROM 'C:\temp\eep.dat'
  WITH
  (
  DATAFILETYPE ='CHAR',
  FORMATFILE='C:\temp\translate_eep.xml',
  ERRORFILE='C:\temp\ERR_eep.TXT',
  FIELDTERMINATOR='0X0A',
  ROWTERMINATOR='0X0A'
  )
GO

Translate_eep.xml looks like this:

<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
 <RECORD>
  <FIELD ID="1"   xsi:type="CharFixed" LENGTH="5"/>
  <FIELD ID="2"   xsi:type="CharFixed" LENGTH="5"/>
  <FIELD ID="3"   xsi:type="CharFixed" LENGTH="5"/>
  <FIELD ID="4"   xsi:type="CharTerm"  TERMINATOR="0X0A"/>
 </RECORD>
 <ROW>
  <COLUMN SOURCE="1"  NAME="c1"  xsi:type="SQLCHAR"/>
  <COLUMN SOURCE="2"  NAME="c2"  xsi:type="SQLCHAR"/>
  <COLUMN SOURCE="3"  NAME="c3"  xsi:type="SQLCHAR"/>
  <COLUMN SOURCE="4"  NAME="c4"  xsi:type="SQLCHAR"/>
 </ROW>
</BCPFORMAT>

Unfortunately, when I use this, it imports one line and then terminates. How can I get this thing to read the entire file?

Sample Data:

ABCDE12345EMILYLove is not all. It is not meat, nor drink, nor slumber<lf>
FGHIJ67890SNL  Oh, no! Mr. bill!<lf>
KLMNO24680ALEX All Nature is but art, unknown to thee<lf>
PQRST13579FROSTSome say the world will end in fire,<lf>

Answer 1

I was unable to solve this problem using BULK INSERT. For the final 3 tables, the ones that had the variable length fields, I used Perl to upload the data with a stored procedure. It looks something like this:

#!/usr/bin/perl

use strict;
use DBI;

# Open the database
my ($dbh,$sth);
my $db_name="MyDB";
my $db_instance="AMACHINE";

$dbh = DBI->connect("DBI:ODBC:Driver={SQL Server};Server=$db_instance;Database=$db_name;") or die "Unable to connect";



my $sql = "DELETE [MyDB].[dbo].[X]" ;
my $sth = $dbh->prepare($sql);
$sth->execute();

my $sql = "{call MyDB.dbo.Import_X_table(?, ?, ?, ?)}" ;
my $sth = $dbh->prepare($sql);

my $in_datafile = "eep.dat";
open INF, $in_datafile || die "Unable to open file $in_datafile\n";

my $lc = 0;  # line counter;
my $buffer;
while ( $buffer = <INF> )
{
  $lc++;
  my $reference = substr($buffer, 0, 5);
  my $code      = substr($buffer, 5, 5);
  my $name      = substr($buffer, 10, 5);
  my $narrative = substr($buffer, 15); #everything after character 14

  $sth->execute($reference, $code, $name, $narrative);

}

The stored procedure is like this:

USE [MyDB]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE PROCEDURE [dbo].[Import_X_table]
    @ent       varchar(5),
    @cd        varchar(5),
    @nm        varchar(5),
    @narr      varchar(max)
AS
BEGIN
    SET NOCOUNT ON
    INSERT INTO [MyDB].[dbo].[X] VALUES (@ent,@cd,@nm, @narr)
END

I don’t like this solution, but it’s the quickest work-around I could come up with.
It get to the objective by inelegantly circumventing the immediate problem rather than solving it.

I don’t whether I should accept this as “The Solution” to this problem.