Question

We have a hardware system with some FPGA’s and an FTDI USB controller. The hardware streams data over USB bulk transfer to the PC at around 5MB/s and the software is tasked with staying in sync, checking the CRC and writing the data to file.

The FTDI chip has a ‘busy’ pin which goes high while its waiting for the PC to do its business. There is a limited amount of buffering in the FTDI and elsewhere on the hardware.

The busy line is going high for longer than the hardware can buffer (50-100ms) so we are losing data. To save us from having to re-design the hardware I have been asked to ‘fix’ this issue!

I think my code is quick enough as we’ve had it running up to 15MB/s, so that leaves an IO bottleneck somewhere. Are we just expecting too much from the PC/OS?

Here is my data entry point. Occasionally we get a dropped bit or byte. If the checksum doesn’t compute, I shift through until it does. byte[] data is nearly always 4k.

    void ftdi_OnData(byte[] data)
    {
        List<byte> buffer = new List<byte>(data.Length);
        int index = 0;

        while ((index + rawFile.Header.PacketLength + 1) < data.Length)
        {
            if (CheckSum.CRC16(data, index, rawFile.Header.PacketLength + 2)) // <- packet length + 2 for 16bit checksum
            {
                buffer.AddRange(data.SubArray<byte>(index, rawFile.Header.PacketLength));                 
                index += rawFile.Header.PacketLength + 2; // <- skip the two checksums, we dont want to save them...
            }
            else
            {
                index++; // shift through
            }
        }

        rawFile.AddData(buffer.ToArray(), 0, buffer.Count);
    }

Answer 1

Tip: do not write to a file…. queue.

Modern computers have multiple processors. If you want certain things as fast as possible, use multiple processors.

• Have on thread deal with the USB data, check checksums etc. It queues (ONLY) the results to a thread safe queue.
• Another thread reads data from the queue and writes it to a file, possibly buffered.

Finished 😉

100ms is a lot of time for decent operations. I have successfully managed around 250.000 IO data packets per second (financial data) using C# without a sweat.

Basically, make sure your IO threads do ONLY that and use your internal memory as buffer. Especially dealing with hardware on one end the thread doing that should ONLY do that, POSSIBLY if needed running in high priority.