Question

this question is a continuation of the first 2 part, anyone who is interested to see where I come from you can refer to part 1 and part 2, but it is not necessary.

write file need to optimised for heavy traffic

Write file need to optimised for heavy traffic part 2

now i have a working snippet, the relevant part is below:

    public static class memoryStreamClass
    {
        static MemoryStream ms1 = new MemoryStream();

        public static void fillBuffer(string outputString)
        {
            byte[] outputByte = Encoding.ASCII.GetBytes(outputString);

            ms1.Write(outputByte, 0, outputByte.Length);

            if (ms1.Length > 8100)
            {
                emptyBuffer(ms1);
                ms1.SetLength(0);
                ms1.Position = 0;
            }
        }

        static void emptyBuffer(MemoryStream ms)
        {
            FileStream outStream = new FileStream("c:\\output.txt", FileMode.Append);

            ms.WriteTo(outStream);
            outStream.Flush();
            outStream.Close();
        }

the above snippet works fine, and bug free. it output around 8KB of data every write.

now i try to multithread the above code to enhance the performance of an IO write bottleneck and problems appeared. the below snippet is what i tried to attempted.

Basically i have 2 identical memoryStream, if say ms1 is full, it writes ms1 into file and switch to ms2 while ms1 is writing, and vice versa.

    public static class memoryStreamClass
    {
        static MemoryStream ms1 = new MemoryStream();
        static MemoryStream ms2 = new MemoryStream();
        static int c = 1;

        public static void fillBuffer(string outputString)
        {
            byte[] outputByte = Encoding.ASCII.GetBytes(outputString);

            if (c == 1)
            {
                ms1.Write(outputByte, 0, outputByte.Length);

                if (ms1.Length > 8100)
                {
                    c = 2;

                    Thread thread1 = new Thread( () => emptyBuffer(ms1));
                    thread1.Start();

                    ms1.SetLength(0);
                    ms1.Position = 0;
                }
            }
            else
            {
                ms2.Write(outputByte, 0, outputByte.Length);

                if (ms2.Length > 8100)
                {
                    c = 1;

                    Thread thread2 = new Thread(() => emptyBuffer(ms2));
                    thread2.Start();

                    ms2.SetLength(0);
                    ms2.Position = 0;

                }
            }
        }

the above code can compile and run, however, the output write is not always 8KB, and it writes way too frequently (than my single thread program). can someone enlighten my and points out what is wrong with my program? thank you very much

Answer 1

Your code is horribly broken, and your idea of using two buffers to improve performance is almost certainly an over-optimisation. However, there’s one obvious problem in this block of code:

Thread thread1 = new Thread( () => emptyBuffer(ms1));
thread1.Start();

ms1.SetLength(0);
ms1.Position = 0;

What this code does is:

• start a thread to process a buffer
• immediately clear that buffer

The problem is that your “clear” code will almost certainly execute before your thread has a chance to start (because generally speaking, an executing method will complete before the thread context changes). So, by the time you call emptyBuffer, your MemoryStream is already empty.

Your statics are a bad idea; if you were to pass a non-static instance to the emptyBuffer method, and then set ms1 = new MemoryStream(), you would probably have better functioning code. But ultimately, this code is conceptually flawed and you should look at redesigning.