Question

I am working on a WPF application that needs to display several video streams at a fast frame rate (we would like 30 fps). The video streams are 1920×1080 raw (RGB24) frames (they are stored in a System.Drawing.Bitmap). Does anyone have any ideas on how to achieve this?

More details:

• Our previous attempts have used a standard WPF Image control, changing its source for each frame. This worked well for a single stream but now that we have to render multiple streams, it is slowing down.
• We have also tried using Direct2D to handle the drawing, using a D3D9 shared surface as the source for an Image control. While this was faster, we are still not able to get a stable 30fps out of it (it jumps between 24-32 fps as things back up).
• The video stream is coming in on a background thread, then being marshaled (using the Dispatcher of the window) to the proper UI thread for drawing. All the drawing is then done on the UI thread. We have also tried giving each window its own thread.

I can provide code samples of what we have tried if anyone wants to see.

thanks!

Answer 1

Anyone looking for a solution, we wrote a custom winforms control using DirectX 11 with a highly optimized copy into graphics memory, then hosted the control in a WinformsHost, I can provide some code to anyone who is interested.

Optimized copy into gpu memory

    // Using native memcpy for the fastest possible copy
    [DllImport("msvcrt.dll", EntryPoint = "memcpy", CallingConvention = CallingConvention.Cdecl, SetLastError = false)]
    private static extern IntPtr memcpy(IntPtr dest, IntPtr src, UIntPtr count);

    /// <summary>
    /// Copies a bitmap to gpu memory
    /// </summary>
    /// <param name="frame">The image to copy to the gpu</param>
    /// <returns>A texture in gpu memory for the bitmap</returns>
    public Texture2D CopyFrameToGpuMemory(Bitmap frame)
    {
        SampleDescription sampleDesc = new SampleDescription();
        sampleDesc.Count = 1;
        sampleDesc.Quality = 0;

        Texture2DDescription texDesc = new Texture2DDescription()
        {
            ArraySize = 1,
            MipLevels = 1,
            SampleDescription = sampleDesc,
            Format = Format.B8G8R8A8_UNorm,
            CpuAccessFlags = CpuAccessFlags.Write,
            BindFlags = BindFlags.ShaderResource,
            Usage = ResourceUsage.Dynamic,
            Height = frame.Height,
            Width = frame.Width
        };

        Texture2D tex = new Texture2D(Device, texDesc);

        Surface surface = tex.AsSurface();
        DataRectangle mappedRect = surface.Map(SlimDX.DXGI.MapFlags.Write | SlimDX.DXGI.MapFlags.Discard);

        BitmapData pixelData = frame.LockBits(new Rectangle(0, 0, frame.Width, frame.Height), ImageLockMode.ReadOnly, PixelFormat.Format32bppRgb);

        unsafe //!!!
        {
            byte* pixelDataStart = (byte*)pixelData.Scan0.ToPointer();
            byte* mappedRectDataStart = (byte*)mappedRect.Data.DataPointer.ToPointer();

            for (int y = 0; y < texDesc.Height; y++)
            {
                byte* lineLocationDest = mappedRectDataStart + (y * mappedRect.Pitch);
                byte* lineLocationSrc = pixelDataStart + (y * pixelData.Stride);

                // Copy line by line for best performance
                memcpy((IntPtr)lineLocationDest, (IntPtr)lineLocationSrc, (UIntPtr)(texDesc.Width * 4));
            }
        } //!!!

        frame.UnlockBits(pixelData);

        mappedRect.Data.Dispose();

        surface.Unmap();
        surface.Dispose();

        return tex;
    }