Question

How do I decode a file with hardware acceleration with ffmpeg?

I have written a working video player that uses ffmpeg.
I have checked for support using "av_hwaccel_next" and found mpeg2_dxva.

However, when I load an mpeg2 file (as usual) I do not get any hardware acceleration. AVCodecContext->hwaccel and AVCodecContext->hwaccelcontext are both null.

Do I have to pass some flag somewhere in order to enable hw-acceleration?

I haven’t been able to find any information regarding this, anyone know of a good source?

Answer 1

Start from reading ffmpeg documentation: https://trac.ffmpeg.org/wiki/HWAccelIntro and better answer How to use hardware acceleration with ffmpeg (and for linux check page https://wiki.archlinux.org/index.php/Hardware_video_acceleration)

When using FFmpeg the tool, HW-assisted decoding is enabled using through the -hwaccel option, which enables a specific decoder. Each decoder may have specific limitations (for example an H.264 decoder may only support baseline profile). HW-assisted encoding is enabled through the use of a specific encoder (for example h264_nvenc). Filtering HW-assisted processing is only supported in a few filters ..
There are several hardware acceleration standards API, some of which are supported to some extent by FFmpeg.

hwaccel activation was controlled by code like (bit reformatted after 2013 https://github.com/FFmpeg/FFmpeg/commit/08303d774132775d49d4ba767092de5d426f089d)

avctx->hwaccel = ff_find_hwaccel(avctx->codec->id, avctx->pix_fmt);

For example, in libavcodec/mpeg12dec.c https://github.com/FFmpeg/FFmpeg/blob/6c7254722ad43712db5686feee8bf75c74d8635b/libavcodec/mpeg12dec.c

avctx->pix_fmt = mpeg_get_pixelformat(avctx);
avctx->hwaccel = ff_find_hwaccel(avctx->codec->id, avctx->pix_fmt);

The ff_find_hwaccel checks codec and pixelformat pair of image and of all available hwaccelerators.

AVHWAccel *ff_find_hwaccel(enum CodecID codec_id, enum PixelFormat pix_fmt)
{
    AVHWAccel *hwaccel=NULL;

    while((hwaccel= av_hwaccel_next(hwaccel))){
        if (   hwaccel->id      == codec_id
            && hwaccel->pix_fmt == pix_fmt)
            return hwaccel;
    }
    return NULL;
}

For example, dxva2 (https://en.wikipedia.org/wiki/DirectX_Video_Acceleration) has:

AVHWAccel mpeg2_dxva2_hwaccel = {
    .name           = "mpeg2_dxva2",
    .type           = AVMEDIA_TYPE_VIDEO,
    .id             = CODEC_ID_MPEG2VIDEO,
    .pix_fmt        = PIX_FMT_DXVA2_VLD,
    .capabilities   = 0,
    .start_frame    = start_frame,
    .decode_slice   = decode_slice,
    .end_frame      = end_frame,
    .priv_data_size = sizeof(struct dxva2_picture_context),
};

And libavutil/pixfmt.h lists all supported hw decoders / accelerators by their pixel formats https://ffmpeg.org/doxygen/3.2/pixfmt_8h.html

AV_PIX_FMT_XVMC_MPEG2_MC    - XVideo Motion Acceleration via common packet passing.
AV_PIX_FMT_XVMC_MPEG2_IDCT  - undocumented
AV_PIX_FMT_XVMC         - undocumented
AV_PIX_FMT_VDPAU_H264   - H.264 HW decoding with VDPAU, data[0] contains a vdpau_render_state struct which contains the bitstream of the slices as well as various fields extracted from headers.
AV_PIX_FMT_VDPAU_MPEG1  - MPEG-1 HW decoding with VDPAU, data[0] contains a vdpau_render_state struct which contains the bitstream of the slices as well as various fields extracted from headers.
AV_PIX_FMT_VDPAU_MPEG2  - MPEG-2 HW decoding with VDPAU, data[0] contains a vdpau_render_state struct which contains the bitstream of the slices as well as various fields extracted from headers.
AV_PIX_FMT_VDPAU_WMV3   - WMV3 HW decoding with VDPAU, data[0] contains a vdpau_render_state struct which contains the bitstream of the slices as well as various fields extracted from headers.
AV_PIX_FMT_VDPAU_VC1    - VC-1 HW decoding with VDPAU, data[0] contains a vdpau_render_state struct which contains the bitstream of the slices as well as various fields extracted from headers. 
AV_PIX_FMT_VAAPI_MOCO   - HW acceleration through VA API at motion compensation entry-point, Picture.data[3] contains a vaapi_render_state struct which contains macroblocks as well as various fields extracted from headers.
AV_PIX_FMT_VAAPI_IDCT   - HW acceleration through VA API at IDCT entry-point, Picture.data[3] contains a vaapi_render_state struct which contains fields extracted from headers.
AV_PIX_FMT_VAAPI_VLD    - HW decoding through VA API, Picture.data[3] contains a VASurfaceID. 
AV_PIX_FMT_VDPAU_MPEG4  - MPEG-4 HW decoding with VDPAU, data[0] contains a vdpau_render_state struct which contains the bitstream of the slices as well as various fields extracted from headers.
AV_PIX_FMT_DXVA2_VLD    - HW decoding through DXVA2, Picture.data[3] contains a LPDIRECT3DSURFACE9 pointer. 
AV_PIX_FMT_VDPAU        - HW acceleration through VDPAU, Picture.data[3] contains a VdpVideoSurface. 
AV_PIX_FMT_VDA          - HW acceleration through VDA, data[3] contains a CVPixelBufferRef. 
AV_PIX_FMT_QSV          - HW acceleration through QSV, data[3] contains a pointer to the mfxFrameSurface1 structure.
AV_PIX_FMT_MMAL         - HW acceleration though MMAL, data[3] contains a pointer to the MMAL_BUFFER_HEADER_T structure.
AV_PIX_FMT_D3D11VA_VLD  - HW decoding through Direct3D11, Picture.data[3] contains a ID3D11VideoDecoderOutputView pointer.
AV_PIX_FMT_CUDA         - HW acceleration through CUDA. data[i] contain CUdeviceptr pointers exactly as for system memory frames. 

Actual selection of pixel formats is in function called before ff_find_hwaccel, the static enum PixelFormat mpeg_get_pixelformat(AVCodecContext *avctx) of libavcodec/mpeg12dec.c for mpeg1/2. In earlier versions of ffmpeg/libavcodec it checks avctx->xvmc_acceleration and/or avctx->codec->capabilities&CODEC_CAP_HWACCEL_VDPAU or calls avctx->get_format(avctx,ff_hwaccel_pixfmt_list_420); to enable hw decoding in some cases.

In recent version (2017) it and several nearby function do the selection of hw coder https://github.com/FFmpeg/FFmpeg/blob/aff8cf18cb0b1fa4f2e3d163c3da2f25aa6d1906/libavcodec/mpeg12dec.c#L1189.

Basically: hardware decoder and its api (obsolete XVMC, VDPAU, VA API, MS DXVA or MS Direct3D11, or videotoolbox) should be enabled in your build of ffmpeg and supported by your hardware and its driver/libraries (many are proprietary and should be downloaded separately). Sometimes -hwaccel option should be given to ffmpeg, or plugin loaded. In linux you may use vainfo and vdpauinfo commands to test availability and supported profiles with most popular standard video hw decoding APIs.

Input file (for mpeg1/2) should be not Grayscale, it should have s->chroma_format less of 2 (4:2:0 Chroma subsampling, which is usual for ISO/IEC MPEG and ITU-T VCEG H.26x; but not for some MPEG-4 Part 2 and not for high 4:4:4 variants of H.264/MPEG-4 AVC).

static const enum AVPixelFormat mpeg2_hwaccel_pixfmt_list_420[] = {
#if CONFIG_MPEG2_XVMC_HWACCEL
    AV_PIX_FMT_XVMC,
#endif
#if CONFIG_MPEG_VDPAU_DECODER && FF_API_VDPAU
    AV_PIX_FMT_VDPAU_MPEG2,
#endif
#if CONFIG_MPEG2_VDPAU_HWACCEL
    AV_PIX_FMT_VDPAU,
#endif
#if CONFIG_MPEG2_DXVA2_HWACCEL
    AV_PIX_FMT_DXVA2_VLD,
#endif
#if CONFIG_MPEG2_D3D11VA_HWACCEL
    AV_PIX_FMT_D3D11VA_VLD,
#endif
#if CONFIG_MPEG2_VAAPI_HWACCEL
    AV_PIX_FMT_VAAPI,
#endif
#if CONFIG_MPEG2_VIDEOTOOLBOX_HWACCEL
    AV_PIX_FMT_VIDEOTOOLBOX,
#endif
    AV_PIX_FMT_YUV420P,
    AV_PIX_FMT_NONE
};

static const enum AVPixelFormat mpeg12_pixfmt_list_422[] = {
    AV_PIX_FMT_YUV422P,
    AV_PIX_FMT_NONE
};

static const enum AVPixelFormat mpeg12_pixfmt_list_444[] = {
    AV_PIX_FMT_YUV444P,
    AV_PIX_FMT_NONE
};

#if FF_API_VDPAU
static inline int uses_vdpau(AVCodecContext *avctx) {
    return avctx->pix_fmt == AV_PIX_FMT_VDPAU_MPEG1 || avctx->pix_fmt == AV_PIX_FMT_VDPAU_MPEG2;
}
#endif

static enum AVPixelFormat mpeg_get_pixelformat(AVCodecContext *avctx)
{
    Mpeg1Context *s1  = avctx->priv_data;
    MpegEncContext *s = &s1->mpeg_enc_ctx;
    const enum AVPixelFormat *pix_fmts;

    if (CONFIG_GRAY && (avctx->flags & AV_CODEC_FLAG_GRAY))
        return AV_PIX_FMT_GRAY8;

    if (s->chroma_format < 2)
        pix_fmts = avctx->codec_id == AV_CODEC_ID_MPEG1VIDEO ?
                                mpeg1_hwaccel_pixfmt_list_420 :
                                mpeg2_hwaccel_pixfmt_list_420;
    else if (s->chroma_format == 2)
        pix_fmts = mpeg12_pixfmt_list_422;
    else
        pix_fmts = mpeg12_pixfmt_list_444;

    return ff_thread_get_format(avctx, pix_fmts);
}

static void setup_hwaccel_for_pixfmt(AVCodecContext *avctx)
{
    // until then pix_fmt may be changed right after codec init
    if (avctx->hwaccel
#if FF_API_VDPAU
        || uses_vdpau(avctx)
#endif
        )
        if (avctx->idct_algo == FF_IDCT_AUTO)
            avctx->idct_algo = FF_IDCT_SIMPLE;

    if (avctx->hwaccel && avctx->pix_fmt == AV_PIX_FMT_XVMC) {
        Mpeg1Context *s1 = avctx->priv_data;
        MpegEncContext *s = &s1->mpeg_enc_ctx;

        s->pack_pblocks = 1;
#if FF_API_XVMC
FF_DISABLE_DEPRECATION_WARNINGS
        avctx->xvmc_acceleration = 2;
FF_ENABLE_DEPRECATION_WARNINGS
#endif /* FF_API_XVMC */
    }
}