Question

I’ve been banging my head all the week trying to serve on my website an mp3 that is being converted.

It seemed really simple.

• I first think: I will redirect the output of ffmpeg process to the web response, then realized that the output is corrupted whatever the encoding I choose: Windows version of ffmpeg stdout just outputs corrupted bytes, that’s weird for an important program like it, but that’s true. Probably an incompatibility with the fact that the Windows console doesn’t output in CP65001. Dunno.

• I then tried to redirect the content of a file written by ffmpeg, but a lot of exceptions were raised, about the end of the stream being reached, or some file permission problem. I’ve read a lot of thread on SO about reading a file being written, but in several cases, people just choose to use another approach, less dirty.

• I tried to use a program coded my NeoSmart, called UTFRedirect.exe, that has been created to allow outputing -REAL- UTF8 content to stdout. It was. But when it lauches the slave process, this program is using ‘WaitForSingleObject’ in C++, so it is not asynchronous. A synchronous method to output content is useless, since in that case, one would use a file and ‘ReadAllBytes’ from it.

• I tried SoX. Same problem with the stdout encoding, and it handles URL very bad.

• I tried: Streaming mp3 while it is being generated on the server. But that’s not similar to my problem, finally.

• I tried to see if I had more chance if I created a file with a pipe-to-file with ffmpeg
  ffmpeg -i input.mp3 -f mp3 - >output.mp3 since it seemed I couldn’t really access the file created by ffmpeg with ffmpeg -i input.mp3 -f mp3 output.mp3.

Is there someone that once faced the same problem?

[EDIT 09/29]

I decided to write my own redirect.exe program so I can redirect anything into it. I made it with two modes: either it writes the bytes into a FileShare.ReadWrite file, either it outputs HEX to the screen so that it is possible to convert it to bytes.

With this solution, I’m 95% sure I’ll be able to reach my goal, which is to use stdout of programs like ffmpeg or sox ‘on the fly’. I’ll probably come back with an answer. But I already know this solution is slow, since representing one hex = two bytes. Maybe I’ll use a another ‘base’ to represent my data, in other words, compress it, so it can be represented in ASCII or anything without null characters, but it takes less space.

For those interested, to read -REAL- bytes from the standard input, there’s this method:

using (Stream stdin = Console.OpenStandardInput())
    using (Stream stdout = Console.OpenStandardOutput())
            {
                byte[] buffer = new byte[2048];
                int bytes;
                while ((bytes = stdin.Read(buffer, 0, buffer.Length)) > 0)
                {                        
                    // Now you have a buffer of bytes in 'buffer'
                }
            }                

[EDIT 09/30]

As I’m now able to get bytes, I’ll output the StandardOutput bytes as Ascii85. It should take 25% more space instead of 100%.

Answer 1

I made a C# console app that redirects a program’s stdout and outputs bytes as ascii85. So no more worries about Windows shell capacity to outputs bytes to a C# process.

I currently use it to redirect on the fly to a webresponse bytes from an MP3 converted in ffmpeg, and it works.

Ascii85 represents a 25% overhead versus bytes space, if you compare it to strings of hexs that represents a 100% overhead, and base64 that represents a 33% overhead on your datasize.

I used Jeff Atwood’s Ascii85 class found on his Coding Horror blog: http://www.codinghorror.com/blog/2005/10/c-implementation-of-ascii85.html

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;

namespace stdshare {
    class Program
    {
        static void Main(string[] args)
        {

                Int32 dur = 0;
        try 
                {
                    dur = Convert.ToInt32(args[0]);
        } catch { return; }

            Console.InputEncoding = Encoding.UTF8;
            Console.OutputEncoding = Encoding.UTF8;

                    using (Stream stdin = Console.OpenStandardInput())
                    using (Stream stdout = Console.OpenStandardOutput())
                    {
                        byte[] buffer = new byte[2048];
                        byte[] buffer2 = new byte[2048];

                        int bytes;
                int pos = 0;
                        using (BinaryReader br = new BinaryReader(stdin))
                        while ((bytes = br.Read(buffer, 0, buffer.Length)) > 0)
                        {
                            buffer2 = new byte[bytes];
                            Buffer.BlockCopy(buffer, 0, buffer2, 0, bytes);
                            string s = System.Convert.ToBase64String(buffer2);
                            Console.WriteLine(s);
                            Console.Out.Flush();
                            stdout.Flush();                            
                  pos += 1;
                  if (pos > dur) break;
                        }


                    }                 



        }


        class Ascii85
        {
            /// <summary>
            /// Prefix mark that identifies an encoded ASCII85 string, traditionally '<~'
            /// </summary>
            public string PrefixMark = "<~";
            /// <summary>
            /// Suffix mark that identifies an encoded ASCII85 string, traditionally '~>'
            /// </summary>
            public string SuffixMark = "~>";
            /// <summary>
            /// Maximum line length for encoded ASCII85 string; 
            /// set to zero for one unbroken line.
            /// </summary>
            public int LineLength = 0;
            /// <summary>
            /// Add the Prefix and Suffix marks when encoding, and enforce their presence for decoding
            /// </summary>
            public bool EnforceMarks = true;

            private const int _asciiOffset = 33;
            private byte[] _encodedBlock = new byte[5];
            private byte[] _decodedBlock = new byte[4];
            private uint _tuple = 0;
            private int _linePos = 0;

            private uint[] pow85 = { 85 * 85 * 85 * 85, 85 * 85 * 85, 85 * 85, 85, 1 };

            /// <summary>
            /// Decodes an ASCII85 encoded string into the original binary data
            /// </summary>
            /// <param name="s">ASCII85 encoded string</param>
            /// <returns>byte array of decoded binary data</returns>
            public byte[] Decode(string s)
            {
                if (EnforceMarks)
                {
                    if (!s.StartsWith(PrefixMark) | !s.EndsWith(SuffixMark))
                    {
                        throw new Exception("ASCII85 encoded data should begin with '" + PrefixMark +
                            "' and end with '" + SuffixMark + "' Problematic string: " + s);
                    }
                }

                // strip prefix and suffix if present
                if (s.StartsWith(PrefixMark))
                {
                    s = s.Substring(PrefixMark.Length);
                }
                if (s.EndsWith(SuffixMark))
                {
                    s = s.Substring(0, s.Length - SuffixMark.Length);
                }

                MemoryStream ms = new MemoryStream();
                int count = 0;
                bool processChar = false;

                foreach (char c in s)
                {
                    switch (c)
                    {
                        case 'z':
                            if (count != 0)
                            {
                                throw new Exception("The character 'z' is invalid inside an ASCII85 block.");
                            }
                            _decodedBlock[0] = 0;
                            _decodedBlock[1] = 0;
                            _decodedBlock[2] = 0;
                            _decodedBlock[3] = 0;
                            ms.Write(_decodedBlock, 0, _decodedBlock.Length);
                            processChar = false;
                            break;
                        case '\n':
                        case '\r':
                        case '\t':
                        case '\0':
                        case '\f':
                        case '\b':
                            processChar = false;
                            break;
                        default:
                            if (c < '!' || c > 'u')
                            {
                                throw new Exception("Bad character '" + c + "' found. ASCII85 only allows characters '!' to 'u'.");
                            }
                            processChar = true;
                            break;
                    }

                    if (processChar)
                    {
                        _tuple += ((uint)(c - _asciiOffset) * pow85[count]);
                        count++;
                        if (count == _encodedBlock.Length)
                        {
                            DecodeBlock();
                            ms.Write(_decodedBlock, 0, _decodedBlock.Length);
                            _tuple = 0;
                            count = 0;
                        }
                    }
                }

                // if we have some bytes left over at the end..
                if (count != 0)
                {
                    if (count == 1)
                    {
                        throw new Exception("The last block of ASCII85 data cannot be a single byte.");
                    }
                    count--;
                    _tuple += pow85[count];
                    DecodeBlock(count);
                    for (int i = 0; i < count; i++)
                    {
                        ms.WriteByte(_decodedBlock[i]);
                    }
                }

                return ms.ToArray();
            }

            /// <summary>
            /// Encodes binary data into a plaintext ASCII85 format string
            /// </summary>
            /// <param name="ba">binary data to encode</param>
            /// <returns>ASCII85 encoded string</returns>
            public string Encode(byte[] ba)
            {
                StringBuilder sb = new StringBuilder((int)(ba.Length * (_encodedBlock.Length / _decodedBlock.Length)));
                _linePos = 0;

                if (EnforceMarks)
                {
                    AppendString(sb, PrefixMark);
                }

                int count = 0;
                _tuple = 0;
                foreach (byte b in ba)
                {
                    if (count >= _decodedBlock.Length - 1)
                    {
                        _tuple |= b;
                        if (_tuple == 0)
                        {
                            AppendChar(sb, 'z');
                        }
                        else
                        {
                            EncodeBlock(sb);
                        }
                        _tuple = 0;
                        count = 0;
                    }
                    else
                    {
                        _tuple |= (uint)(b << (24 - (count * 8)));
                        count++;
                    }
                }

                // if we have some bytes left over at the end..
                if (count > 0)
                {
                    EncodeBlock(count + 1, sb);
                }

                if (EnforceMarks)
                {
                    AppendString(sb, SuffixMark);
                }
                return sb.ToString();
            }

            private void EncodeBlock(StringBuilder sb)
            {
                EncodeBlock(_encodedBlock.Length, sb);
            }

            private void EncodeBlock(int count, StringBuilder sb)
            {
                for (int i = _encodedBlock.Length - 1; i >= 0; i--)
                {
                    _encodedBlock[i] = (byte)((_tuple % 85) + _asciiOffset);
                    _tuple /= 85;
                }

                for (int i = 0; i < count; i++)
                {
                    char c = (char)_encodedBlock[i];
                    AppendChar(sb, c);
                }

            }

            private void DecodeBlock()
            {
                DecodeBlock(_decodedBlock.Length);
            }

            private void DecodeBlock(int bytes)
            {
                for (int i = 0; i < bytes; i++)
                {
                    _decodedBlock[i] = (byte)(_tuple >> 24 - (i * 8));
                }
            }

            private void AppendString(StringBuilder sb, string s)
            {
                if (LineLength > 0 && (_linePos + s.Length > LineLength))
                {
                    _linePos = 0;
                    sb.Append('\n');
                }
                else
                {
                    _linePos += s.Length;
                }
                sb.Append(s);
            }

            private void AppendChar(StringBuilder sb, char c)
            {
                sb.Append(c);
                _linePos++;
                if (LineLength > 0 && (_linePos >= LineLength))
                {
                    _linePos = 0;
                    sb.Append('\n');
                }
            }

        }
    }
}