High Efficiency Video Coding (HEVC), also known as H.265 and MPEG-H Part 2, is a patent-encumbered video compression standard designed as part of the MPEG-H project as a successor to the widely used Advanced Video Coding (AVC, H.264, or MPEG-4 Part 10). The standard was published in 2013. In comparison to AVC, HEVC can be used to achieve from 25% to 50% better data compression at the same level of video quality, or substantially improved video quality at the same bit rate. For example, standard 4K video capture at 30 frames per second typically requires a target bitrate of approximately 50 Mbps using HEVC, requiring roughly 22.5 GB of storage volume per hour, compared to the 100 Mbps and 45 GB per hour required by older H.264 profiles. It supports resolutions up to 8192×4320, including 8K UHD, and unlike the primarily eight-bit AVC, HEVC's higher-fidelity Main 10 profile has been incorporated into nearly all supporting hardware. The High Efficiency Image Format (HEIF) is a container format whose default codec is HEVC.
While AVC uses the integer discrete cosine transform (DCT) with 4×4 and 8×8 block sizes, HEVC uses both integer DCT and discrete sine transform (DST) with varied block sizes between 4×4 and 32×32.
Concept In most ways, HEVC is an extension of the concepts in H.264/MPEG-4 AVC. Both work by comparing different parts of a frame of video to find areas that are redundant, both within a single frame and between consecutive frames. These redundant areas are then replaced with a short description instead of the original pixels. The primary changes for HEVC include the expansion of the pattern comparison and difference-coding areas from 16×16 pixel to sizes up to 64×64, improved variable-block-size segmentation, improved "intra" prediction within the same picture, improved motion vector prediction and motion region merging, improved motion compensation filtering, and an additional filtering step called sample-adaptive offset filtering. Effective use of these improvements requires much more signal processing capability for compressing the video but has less impact on the amount of computation needed for decompression.
HEVC was standardized by the Joint Collaborative Team on Video Coding (JCT-VC), a collaboration between the ISO/IEC MPEG and ITU-T Study Group 16 VCEG. The ISO/IEC group refers to it as MPEG-H Part 2 and the ITU-T as H.265. The first version of the HEVC standard was ratified in January 2013 and published in June 2013. The second version, with multiview extensions (MV-HEVC), range extensions (RExt), and scalability extensions (SHVC), was completed and approved in 2014 and published in early 2015. Extensions for 3D video (3D-HEVC) were completed in early 2015, and extensions for screen content coding (SCC) were completed in early 2016 and published in early 2017, covering video containing rendered graphics, text, or animation as well as (or instead of) camera-captured video scenes. In October 2017, the standard was recognized by a Primetime Emmy Engineering Award as having had a material effect on the technology of television.
HEVC contains technologies covered by patents owned by the organizations that participated in the JCT-VC. Implementing a device or software application that uses HEVC may require a license from HEVC patent holders. The ISO/IEC and ITU require companies that belong to their organizations to offer their patents on reasonable and non-discriminatory licensing (RAND) terms. Patent licenses can be obtained directly from each patent holder, or through patent licensing bodies, such as MPEG LA, Access Advance, and Velos Media.
The combined licensing fees currently offered by all of the patent licensing bodies are higher than for AVC. The licensing fees are one of the main reasons HEVC adoption has been low on the web and is why some of the largest tech companies (Amazon, AMD, Apple, ARM, Cisco, Google, Intel, Microsoft, Mozilla, Netflix, Nvidia, and more) have joined the Alliance for Open Media, which finalized royalty-free alternative video coding format AV1 on March 28, 2018.
History The HEVC format was jointly developed by more than a dozen organisations across the world. The majority of active patent contributions towards the development of the HEVC format came from five organizations: Samsung Electronics (4,249 patents), General Electric (1,127 patents), M&K Holdings (907 patents), NTT ( patents), and JVC Kenwood (628 patents). Other patent holders include Fujitsu, Apple, Canon, Columbia University, KAIST, Kwangwoon University, MIT, Sungkyunkwan University, Funai, Hikvision, KBS, KT, and NEC.
Previous work In 2004, the ITU-T Video Coding Experts Group (VCEG) began a major study of technology advances that could enable the creation of a new video compression standard (or substantial compression-oriented enhancements of the H.264/MPEG-4 AVC standard). In October 2004, various techniques for potential enhancement of the H.264/MPEG-4 AVC standard were surveyed. In January 2005, at the next meeting of VCEG, VCEG began designating certain topics as "Key Technical Areas" (KTA) for further investigation. A software codebase called the KTA codebase was established for evaluating such proposals.T. Wedi and T. K. Tan, AHG report – Coding Efficiency Improvements, VCEG document VCEG-AA06, 17–18 October 2005. The KTA software was based on the Joint Model (JM) reference software that was developed by the MPEG & VCEG Joint Video Team for H.264/MPEG-4 AVC. Additional proposed technologies were integrated into the KTA software and tested in experiment evaluations over the next four years.Meeting Report for 31st VCEG Meeting VCEG document VCEG-AE01r1, Marrakech, MA, 15–16 January 2007
Two approaches for standardizing enhanced compression technology were considered: either creating a new standard or creating extensions of H.264/MPEG-4 AVC. The project had tentative names H.265 and H.NGVC (Next-generation Video Coding), and was a major part of the work of VCEG until it evolved into the HEVC joint project with MPEG in 2010.
The preliminary requirements for NGVC were the capability to have a bit rate reduction of 50% at the same subjective image quality compared with the H.264/MPEG-4 AVC High profile, and computational complexity ranging from 1/2 to 3 times that of the High profile. NGVC would be able to provide 25% bit rate reduction along with 50% reduction in complexity at the same perceived video quality as the High profile, or to provide greater bit rate reduction with somewhat higher complexity.
The ISO/IEC Moving Picture Experts Group (MPEG) started a similar project in 2007, tentatively named High-performance Video Coding. An agreement of getting a bit rate reduction of 50% had been decided as the goal of the project by July 2007. Early evaluations were performed with modifications of the KTA reference software encoder developed by VCEG. By July 2009, experimental results showed average bit reduction of around 20% compared with AVC High Profile; these results prompted MPEG to initiate its standardization effort in collaboration with VCEG.
Joint Collaborative Team on Video Coding
MPEG and VCEG established a Joint Collaborative Team on Video Coding (JCT-VC) to develop the HEVC standard.
Standardization A formal joint Call for Proposals on video compression technology was issued in January 2010 by VCEG and MPEG, and proposals were evaluated at the first meeting of the MPEG & VCEG Joint Collaborative Team on Video Coding (JCT-VC), which took place in April 2010. A total of 27 full proposals were submitted. Evaluations showed that some proposals could reach the same visual quality as AVC at only half the bit rate in many of the test cases, at the cost of 2–10× increase in computational complexity, and some proposals achieved good subjective quality and bit rate results with lower computational complexity than the reference AVC High profile encodings. At that meeting, the name High Efficiency Video Coding (HEVC) was adopted for the joint project. Starting at that meeting, the JCT-VC integrated features of some of the best proposals into a single software codebase and a "Test Model under Consideration", and performed further experiments to evaluate various proposed features. The first working draft specification of HEVC was produced at the third JCT-VC meeting in October 2010. Many changes in the coding tools and configuration of HEVC were made in later JCT-VC meetings.
On January 25, 2013, the ITU announced that HEVC had received first stage approval (consent) in the ITU-T Alternative Approval Process (AAP). On the same day, MPEG announced that HEVC had been promoted to Final Draft International Standard (FDIS) status in the MPEG standardization process.
On April 13, 2013, HEVC/H.265 was approved as an ITU-T standard. The standard was formally published by the ITU-T on June 7, 2013, and by the ISO/IEC on November 25, 2013.
On July 11, 2014, MPEG announced that the 2nd edition of HEVC will contain three recently completed extensions which are the multiview extensions (MV-HEVC), the range extensions (RExt), and the scalability extensions (SHVC).
On October 29, 2014, HEVC/H.265 version 2 was approved as an ITU-T standard. It was then formally published on January 12, 2015.
On April 29, 2015, HEVC/H.265 version 3 was approved as an ITU-T standard.
On June 3, 2016, HEVC/H.265 version 4 was consented in the ITU-T and was not approved during a vote in October 2016.
On December 22, 2016, HEVC/H.265 version 4 was approved as an ITU-T standard.
Patent licensing
On September 29, 2014, MPEG LA announced their HEVC license which covers the essential patents from 23 companies. The first 100,000 "devices" (which includes software implementations) are royalty-free, and after that the fee is $0.20 per device up to an annual cap of $25 million. This is significantly more expensive than the fees on AVC, which were $0.10 per device, with the same 100,000 waiver, and an annual cap of $6.5 million. MPEG LA does not charge any fee on the content itself, something they had attempted when initially licensing AVC, but subsequently dropped when content producers refused to pay it. The license has been expanded to include the profiles in version 2 of the HEVC standard.
When the MPEG LA terms were announced, commenters noted that a number of prominent patent holders were not part of the group. Among these were AT&T, Microsoft, Nokia, and Motorola. Speculation at the time was that these companies would form their own licensing pool to compete with or add to the MPEG LA pool. Such a group was formally announced on March 26, 2015, as HEVC Advance. The terms, covering 500 essential patents, were announced on July 22, 2015, with rates that depend on the country of sale, type of device, HEVC profile, HEVC extensions, and HEVC optional features. Unlike the MPEG LA terms, HEVC Advance reintroduced license fees on content encoded with HEVC, through a revenue sharing fee.
The initial HEVC Advance license had a maximum royalty rate of US$2.60 per device for Region 1 countries and a content royalty rate of 0.5% of the revenue generated from HEVC video services. Region 1 countries in the HEVC Advance license include the United States, Canada, European Union, Japan, South Korea, Australia, New Zealand, and others. Region 2 countries are countries not listed in the Region 1 country list. The HEVC Advance license had a maximum royalty rate of US$1.30 per device for Region 2 countries. Unlike MPEG LA, there was no annual cap. On top of this, HEVC Advance also charged a royalty rate of 0.5% of the revenue generated from video services encoding content in HEVC.