Question

I am using CSVReader in my project for reading .csv files in windows service.
we pass the .csv file to CSVReader for processing the file and it was working fine.
But recently we decided to save the csv file to Database table and read it from there.
when a user submits an csv file for processing , our aspx page will read the csv file
and converts it to byte array and saves it to database.
and the service will read the table and picks up the byte array and converts it to Filestream.
This stream is passed to CSVReader for further work.
Now it throws error for the last row last column.
Its happening only after saving and reading from Database.
I am getting the following error.
No idea how to fix this.

“The CSV appears to be corrupt near record ‘9’ field ‘3 at position ‘494’. Current raw data : ‘

Hear is the code
Converting file to Byte Array …..

try
{
     fs = new FileStream(filepath, FileMode.Open);
     fsbuffer = new byte[fs.Length];
     fs.Read(fsbuffer, 0, (int)fs.Length);
}

Reading from DB to Byte Array ……

myobject.FileByteArray = ObjectToByteArray(row);

public byte[] ObjectToByteArray(DataRow row)
        {
            if (row["fileBytearray"] == null)
                return null;
            try
            {
                BinaryFormatter bf = new BinaryFormatter();
                System.IO.MemoryStream ms = new System.IO.MemoryStream();

                bf.Serialize(ms, row["fileBytearray"]);
                return ms.ToArray();
            }
    }

Stream fileStream = new MemoryStream(myobject.FileByteArray)
using (CsvReader csv = 
                        new CsvReader(new StreamReader(fileStream, System.Text.Encoding.UTF7), hasHeader, ','))

Answer 1

I haven’t been able to recreate your issue, so instead, as, without seeing the save and load from beginning to end, I’ll suggest trying the following for retrieval:

public MemoryStream LoadReportData(int rowId)
{

  MemoryStream stream = new MemoryStream();

  using (BinaryWriter writer = new BinaryWriter(stream))
  {

    using (DbConnection connection = db.CreateConnection())
    {
      DbCommand selectCommand = "SELECT CSVData FROM YourTable WHERE Id = @rowId";

      selectCommand.Connection = connection;

      db.AddInParameter(selectCommand, "@rowId", DbType.Int32, rowId);
      connection.Open();

      using (IDataReader reader = selectCommand.ExecuteReader(CommandBehavior.SequentialAccess))
      {

        while (reader.Read())
        {

          int startIndex = 0;
          int bufferSize = 8192;

          byte[] buffer = new byte[bufferSize];

          long retVal = reader.GetBytes(0, startIndex, buffer, 0, bufferSize);

          while (retVal == bufferSize)
          {

            writer.Write(buffer);
            writer.Flush();

            startIndex += bufferSize;
            retVal = reader.GetBytes(0, startIndex, buffer, 0, bufferSize);

          }

          writer.Write(buffer, 0, (int)retVal);
        }

      }

    }
  }

  return stream;

}

You’ll need to replace the selectCommand sql and parameters with whatever sql you are using to return the data.

This method uses a SqlDataReader to sequentially read bytes from the appropriate row (identified by rowId in my example) and column (called CSVData in my example) which should avoid any truncation issues on the way out (it could be that your DataTable object is only returning the first n bytes). The MemoryStream object could be used to resave the CSV file to the file system for testing, or fed straight into your CSVReader.

If you can post your Save method (where you actually persist the data to the database) then we can check that too to make sure that truncation isn’t happening there, either.

One other suggestion I can make right now involves your loading the file into a byte array. If you are going to load the file in one go, then you can simply replace:

try
{
     fs = new FileStream(filepath, FileMode.Open);
     fsbuffer = new byte[fs.Length];
     fs.Read(fsbuffer, 0, (int)fs.Length);
}

with

byte[] fileBytes = File.ReadAllBytes(filepath);