Introduction
The OpenVINO™ toolkit is composed of a variety of Intel tools that work together to provide a complete computer vision pipeline solution that is optimized on Intel® architecture. This article will focus on the Intel® Media SDK component of the toolkit. The Intel Media SDK is a high level API for specific video processing operations: decode, process, and encode. It supports H.265, H.264, MPEG-2, and more codecs. Video processing can be used to resize, scale, de-interlace, color conversion, de-noise, sharpen, and more. The Intel Media SDK works in the background to leverage hardware acceleration on Intel® architecture with optimized software fallback for each individual hardware platform. Hence, developers do not need to change the code from platform to platform, and can focus more on the application itself rather than hardware optimization.
To learn more about OpenVINO toolkit:
To learn more about the Intel Media SDK:
Intel® Media SDK Set-Up
OpenVINO toolkit 2018 R1.2 can be downloaded for free here: OpenVINO™ Toolkit Choose & Download.
Instructions on how to install on Ubuntu* 16.04.3 are described here: Installing the OpenVINO™ Toolkit for Linux*.
OpenVINO toolkit will install the Intel Media SDK as well in /opt/intel/mediasdk/
.
To verify the install:
Check the environment variables are correct. MFX_HOME
, LIBVA_DRIVERS_PATH
, and LIBVA_DRIVER_NAME
should be set as seen below.
Code snippet 1. Check the environment variables
Otherwise set the environment variables:
Code snippet 2. Set the environment variables if needed
Install vainfo
:
Code snippet 3. install vainfo
Execute the command vainfo
in a command prompt to ensure it can find the iHD_drv_video.so
and there are no other errors.
Code snippet 4. vainfo
command output
If there are errors on vainfo
, you will need to download the below script and run it as a normal user. For example, you might see "libva error: /opt/intel/mediasdk/lib64/iHD_drv_video.so has no function __vaDriverInit_0_32" in this case, the below script will help fix the media driver install.
Code snippet 5. Install Media SDK dependencies
The script will prompt for three things: enter the sudo password when asked, hit enter when asked to input the working directory so it uses the defaults, and input "Yes" to check out the tested commit.
It is recommended to install a YUV capable video player to view the videos, e.g. vooya* and install with sudo dpkg –i <deb file>
.
Build the Samples
Media SDK comes with a number of sample programs for decode, process, encode, and others. To run them, the samples need to be built first:
Code snippet 6. Build the Media SDK samples
The compiled samples can be found here:
cd/opt/intel/mediasdk/samples/samples/__cmake/intel64.make.debug/__bin/debug
Code snippet 7. location of the compiled samples
To use the samples, move them out of the /opt/
directory, somewhere on the Desktop for example. That way it doesn't require sudo to run, and sudo user does not have the proper environment variables set up.
Video Input
In this paper we will focus on H.264 format as most webcams output video in H.264 format. However the video for this article will start in mp4 format as that is more commonly used and available on the internet. Hence we need to transcode the video from mp4 to H.264. Below will outline two ways to accomplish this, one accelerated version and one not.
Cars driving on the road mp4 video.
Intel® Quick Sync Video Transcode Acceleration
An Intel® Processor with integrated graphics will allow you to take advantage of Intel® Quick Sync Video. Intel Quick Sync Video provides a dedicated media processing capabilities inside of Intel® Processor Graphics technology. To check the processor capabilities visit: Product Specifications
Copy the SDK header files to include/mfx
:
Code snippet 8. Copy the Media SDK header files
Install the dependencies:
Code snippet 9. Install the dependencies
Create the "libmfx.pc" file, inside a folder on the Desktop is fine.
Code snippet 10. libmfx.pc file
Updated the PKG_CONFIG_PATH
environment variable with the path to the libmfx.pc
file:
Code snippet 11. export PKG_CONFIG_PATH
environment variable
Get the FFMPEG source code and build it:
Code snippet 12. build ffmpeg
Check that the codecs are installed properly:
Code snippet 13. check Intel® Quick Sync Video codecs installed
Figure 1. shows Intel® Quick Sync Video codecs installed
Convert the video using the Intel® Quick Sync Video accelerator. You can ignore the info message about "A decode call did not consume any data: expect more data at input", that just indicates it has reached the end of the file.
Code snippet 14. convert mp4 video to h264 format with Intel® Quick Sync Video acceleration
Transcode Without Intel® Quick Sync Video
To transcode the video without Intel® Quick Sync Video, install "ffmpeg" and the "libx264-dev" codec.
Code snippet 15. convert mp4 video to h264 format without Intel Quick Sync Video acceleration
Functions
We will focus on the three main operations the SDK optimizes and comes with sample code for: Decode, VPP (Video Processing Pipeline), and Encode.
Figure 2. Intel® Media SDK pipeline
Decode
The decoding sample supports the following input video formats: H.264 (AVC, MVC – Multi-View Coding), MPEG-2 video, VC-1, JPEG/Motion JPEG, HEVC, VP8). It will render a YUV 4:2:0 video as output. Decode can also resize the video at the same time, just change the width and height parameters. Do not leave the width and height parameters out of the command, even if not resizing.
Example decode command:
VPP (Video Processing Pipeline)
To do some video processing after decoding, the VPP sample supports to following features:
VPP functions:
- N:1 Frame Composition
- Resizing
- Color Conversion
- Deinterlacing
- Denoising
- Frame Rate Conversion
- Brightness/Contrast/Saturation
- Sharpening
Example resizing the video:
Code snippet 17. vpp command
The istab
option will scale the image to fit the new size, otherwise it will crop it to the new size by default.
Encode
Encode sample will encode the video back to its original format or to another format that is supported by decode as input.
Code snippet 18. encode command
Summary
This article has been an introduction to the Intel Media SDK and how to leverage it to accelerate the video workloads for decoding, processing, and encoding a video. It can be great addition to an existing computer vision pipeline to process videos. While inference is normally the largest percentage of the processing time, that time is reduced using the Intel® Deep Learning Deployment Toolkit model optimizer. Hence the time to process the video becomes a second optimization that is worth exploring to fully optimize the computer vision pipeline.
About the author
Whitney Foster is a software engineer at Intel in the Core and Visual Computing Group working on scale enabling projects for Internet of Things, Intel® vPro™ technology, and OpenVINO toolkit.
"