1*77c1e3ccSAndroid Build Coastguard Worker<div style="font-size:3em; text-align:center;"> Algorithm Description </div> 2*77c1e3ccSAndroid Build Coastguard Worker 3*77c1e3ccSAndroid Build Coastguard Worker# Abstract 4*77c1e3ccSAndroid Build Coastguard WorkerThis document describes technical aspects of coding tools included in 5*77c1e3ccSAndroid Build Coastguard Workerthe associated codec. This document is not a specification of the associated 6*77c1e3ccSAndroid Build Coastguard Workercodec. Instead, it summarizes the highlighted features of coding tools for new 7*77c1e3ccSAndroid Build Coastguard Workerdevelopers. This document should be updated when significant new normative 8*77c1e3ccSAndroid Build Coastguard Workerchanges have been integrated into the associated codec. 9*77c1e3ccSAndroid Build Coastguard Worker 10*77c1e3ccSAndroid Build Coastguard Worker# Table of Contents 11*77c1e3ccSAndroid Build Coastguard Worker 12*77c1e3ccSAndroid Build Coastguard Worker[Abbreviations](#Abbreviations) 13*77c1e3ccSAndroid Build Coastguard Worker 14*77c1e3ccSAndroid Build Coastguard Worker[Algorithm description](#Algorithm-Description) 15*77c1e3ccSAndroid Build Coastguard Worker 16*77c1e3ccSAndroid Build Coastguard Worker- [Block Partitioning](#Block-Partitioning) 17*77c1e3ccSAndroid Build Coastguard Worker - [Coding block partition](#Coding-block-partition) 18*77c1e3ccSAndroid Build Coastguard Worker - [Transform block partition](#Transform-block-partition) 19*77c1e3ccSAndroid Build Coastguard Worker- [Intra Prediction](#Intra-Prediction) 20*77c1e3ccSAndroid Build Coastguard Worker - [Directional intra prediction modes](#Directional-intra-prediction-modes) 21*77c1e3ccSAndroid Build Coastguard Worker - [Non-directional intra prediction modes](#Non-directional-intra-prediction-modes) 22*77c1e3ccSAndroid Build Coastguard Worker - [Recursive filtering modes](#Recursive-filtering-modes) 23*77c1e3ccSAndroid Build Coastguard Worker - [Chroma from Luma mode](#Chroma-from-Luma-mode) 24*77c1e3ccSAndroid Build Coastguard Worker- [Inter Prediction](#Inter-Prediction) 25*77c1e3ccSAndroid Build Coastguard Worker - [Motion vector prediction](#Motion-vector-prediction) 26*77c1e3ccSAndroid Build Coastguard Worker - [Motion vector coding](#Motion-vector-coding) 27*77c1e3ccSAndroid Build Coastguard Worker - [Interpolation filter for motion compensation](#Interpolation-filter-for-motion-compensation) 28*77c1e3ccSAndroid Build Coastguard Worker - [Warped motion compensation](#Warped-motion-compensation) 29*77c1e3ccSAndroid Build Coastguard Worker - [Overlapped block motion compensation](#Overlapped-block-motion-compensation) 30*77c1e3ccSAndroid Build Coastguard Worker - [Reference frames](#Reference-frames) 31*77c1e3ccSAndroid Build Coastguard Worker - [Compound Prediction](#Compound-Prediction) 32*77c1e3ccSAndroid Build Coastguard Worker- [Transform](#Transform) 33*77c1e3ccSAndroid Build Coastguard Worker- [Quantization](#Quantization) 34*77c1e3ccSAndroid Build Coastguard Worker- [Entropy Coding](#Entropy-Coding) 35*77c1e3ccSAndroid Build Coastguard Worker- [Loop filtering and post-processing](#Loop-filtering-and-post-processing) 36*77c1e3ccSAndroid Build Coastguard Worker - [Deblocking](#Deblocking) 37*77c1e3ccSAndroid Build Coastguard Worker - [Constrained directional enhancement](#Constrained-directional-enhancement) 38*77c1e3ccSAndroid Build Coastguard Worker - [Loop Restoration filter](#Loop-Restoration-filter) 39*77c1e3ccSAndroid Build Coastguard Worker - [Frame super-resolution](#Frame-super-resolution) 40*77c1e3ccSAndroid Build Coastguard Worker - [Film grain synthesis](#Film-grain-synthesis) 41*77c1e3ccSAndroid Build Coastguard Worker- [Screen content coding](#Screen-content-coding) 42*77c1e3ccSAndroid Build Coastguard Worker - [Intra block copy](#Intra-block-copy) 43*77c1e3ccSAndroid Build Coastguard Worker - [Palette mode](#Palette-mode) 44*77c1e3ccSAndroid Build Coastguard Worker 45*77c1e3ccSAndroid Build Coastguard Worker[References](#References) 46*77c1e3ccSAndroid Build Coastguard Worker 47*77c1e3ccSAndroid Build Coastguard Worker# Abbreviations 48*77c1e3ccSAndroid Build Coastguard Worker 49*77c1e3ccSAndroid Build Coastguard WorkerCfL: Chroma from Luma\ 50*77c1e3ccSAndroid Build Coastguard WorkerIntraBC: Intra block copy\ 51*77c1e3ccSAndroid Build Coastguard WorkerLCU: Largest coding unit\ 52*77c1e3ccSAndroid Build Coastguard WorkerOBMC: Overlapped Block Motion Compensation\ 53*77c1e3ccSAndroid Build Coastguard WorkerCDEF: Constrained Directional Enhancement Filter 54*77c1e3ccSAndroid Build Coastguard Worker 55*77c1e3ccSAndroid Build Coastguard Worker# Algorithm Description 56*77c1e3ccSAndroid Build Coastguard Worker 57*77c1e3ccSAndroid Build Coastguard Worker## Block Partitioning 58*77c1e3ccSAndroid Build Coastguard Worker 59*77c1e3ccSAndroid Build Coastguard Worker### Coding block partition 60*77c1e3ccSAndroid Build Coastguard Worker 61*77c1e3ccSAndroid Build Coastguard WorkerThe largest coding block unit (LCU) applied in this codec is 128×128. In 62*77c1e3ccSAndroid Build Coastguard Workeraddition to no split mode `PARTITION_NONE`, the partition tree supports 9 63*77c1e3ccSAndroid Build Coastguard Workerdifferent partitioning patterns, as shown in below figure. 64*77c1e3ccSAndroid Build Coastguard Worker 65*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\partition_codingblock.svg" 66*77c1e3ccSAndroid Build Coastguard Workeralt="Partition" width="360" /> <figcaption>Figure 1: Supported coding block 67*77c1e3ccSAndroid Build Coastguard Workerpartitions</figcaption> </figure> 68*77c1e3ccSAndroid Build Coastguard Worker 69*77c1e3ccSAndroid Build Coastguard WorkerAccording to the number of sub-partitions, the 9 partition modes are summarized 70*77c1e3ccSAndroid Build Coastguard Workeras follows: 1. Four partitions: `PARTITION_SPLIT`, `PARTITION_VERT_4`, 71*77c1e3ccSAndroid Build Coastguard Worker`PARTITION_HORZ_4` 2. Three partitions (T-Shape): `PARTITION_HORZ_A`, 72*77c1e3ccSAndroid Build Coastguard Worker`PARTITION_HORZ_B`, `PARTITION_VERT_A`, `PARTITION_HORZ_B` 3. Two partitions: 73*77c1e3ccSAndroid Build Coastguard Worker`PARTITION_HORZ`, `PARTITION_VERT` 74*77c1e3ccSAndroid Build Coastguard Worker 75*77c1e3ccSAndroid Build Coastguard WorkerAmong all the 9 partitioning patterns, only `PARTITION_SPLIT` mode supports 76*77c1e3ccSAndroid Build Coastguard Workerrecursive partitioning, i.e., sub-partitions can be further split, other 77*77c1e3ccSAndroid Build Coastguard Workerpartitioning modes cannot further split. Particularly, for 8x8 and 128x128, 78*77c1e3ccSAndroid Build Coastguard Worker`PARTITION_VERT_4`, `PARTITION_HORZ_4` are not used, and for 8x8, T-Shape 79*77c1e3ccSAndroid Build Coastguard Workerpartitions are not used either. 80*77c1e3ccSAndroid Build Coastguard Worker 81*77c1e3ccSAndroid Build Coastguard Worker### Transform block partition 82*77c1e3ccSAndroid Build Coastguard Worker 83*77c1e3ccSAndroid Build Coastguard WorkerFor both intra and inter coded blocks, the coding block can be further 84*77c1e3ccSAndroid Build Coastguard Workerpartitioned into multiple transform units with the partitioning depth up to 2 85*77c1e3ccSAndroid Build Coastguard Workerlevels. The mapping from the transform size of the current depth to the 86*77c1e3ccSAndroid Build Coastguard Workertransform size of the next depth is shown in the following Table 1. 87*77c1e3ccSAndroid Build Coastguard Worker 88*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><figcaption>Table 1: Transform partition size 89*77c1e3ccSAndroid Build Coastguard Workersetting</figcaption> <img src="img\tx_partition.svg" alt="Partition" width="220" 90*77c1e3ccSAndroid Build Coastguard Worker/> </figure> 91*77c1e3ccSAndroid Build Coastguard Worker 92*77c1e3ccSAndroid Build Coastguard WorkerFurthermore, for intra coded blocks, the transform partition is done in a way 93*77c1e3ccSAndroid Build Coastguard Workerthat all the transform blocks have the same size, and the transform blocks are 94*77c1e3ccSAndroid Build Coastguard Workercoded in a raster scan order. An example of the transform block partitioning for 95*77c1e3ccSAndroid Build Coastguard Workerintra coded block is shown in the Figure 2. 96*77c1e3ccSAndroid Build Coastguard Worker 97*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\intra_tx_partition.svg" 98*77c1e3ccSAndroid Build Coastguard Workeralt="Partition" width="600" /> <figcaption>Figure 2: Example of transform 99*77c1e3ccSAndroid Build Coastguard Workerpartitioning for intra coded block</figcaption> </figure> 100*77c1e3ccSAndroid Build Coastguard Worker 101*77c1e3ccSAndroid Build Coastguard WorkerFor inter coded blocks, the transform unit partitioning can be done in a 102*77c1e3ccSAndroid Build Coastguard Workerrecursive manner with the partitioning depth up to 2 levels. The transform 103*77c1e3ccSAndroid Build Coastguard Workerpartitioning supports 1:1 (square), 1:2/2:1, and 1:4/4:1 transform unit sizes 104*77c1e3ccSAndroid Build Coastguard Workerranging from 4×4 to 64×64. If the coding block is smaller than or equal to 105*77c1e3ccSAndroid Build Coastguard Worker64x64, the transform block partitioning can only apply to luma component, for 106*77c1e3ccSAndroid Build Coastguard Workerchroma blocks, the transform block size is identical to the coding block size. 107*77c1e3ccSAndroid Build Coastguard WorkerOtherwise, if the coding block width or height is greater than 64, then both the 108*77c1e3ccSAndroid Build Coastguard Workerluma and chroma coding blocks will implicitly split into multiples of min(W, 109*77c1e3ccSAndroid Build Coastguard Worker64)x min(H, 64) and min(W, 32)x min(H, 32) transform blocks, respectively. 110*77c1e3ccSAndroid Build Coastguard Worker 111*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\inter_tx_partition.svg" 112*77c1e3ccSAndroid Build Coastguard Workeralt="Partition" width="400" /> <figcaption>Figure 3: Example of transform 113*77c1e3ccSAndroid Build Coastguard Workerpartitioning for inter coded block</figcaption> </figure> 114*77c1e3ccSAndroid Build Coastguard Worker 115*77c1e3ccSAndroid Build Coastguard Worker## Intra Prediction 116*77c1e3ccSAndroid Build Coastguard Worker 117*77c1e3ccSAndroid Build Coastguard Worker### Directional intra prediction modes 118*77c1e3ccSAndroid Build Coastguard Worker 119*77c1e3ccSAndroid Build Coastguard WorkerDirectional intra prediction modes are applied in intra prediction, which models 120*77c1e3ccSAndroid Build Coastguard Workerlocal textures using a given direction pattern. Directional intra prediction 121*77c1e3ccSAndroid Build Coastguard Workermodes are represented by nominal modes and angle delta. The nominal modes are 122*77c1e3ccSAndroid Build Coastguard Workersimilar set of intra prediction angles used in VP9, which includes 8 angles. The 123*77c1e3ccSAndroid Build Coastguard Workerindex value of angle delta is ranging from -3 ~ +3, and zero delta angle 124*77c1e3ccSAndroid Build Coastguard Workerindicates a nominal mode. The prediction angle is represented by a nominal intra 125*77c1e3ccSAndroid Build Coastguard Workerangle plus an angle delta. In total, there are 56 directional intra prediction 126*77c1e3ccSAndroid Build Coastguard Workermodes, as shown in the following figure. In the below figure, solid arrows 127*77c1e3ccSAndroid Build Coastguard Workerindicate directional intra prediction modes and dotted arrows represent non-zero 128*77c1e3ccSAndroid Build Coastguard Workerangle delta. 129*77c1e3ccSAndroid Build Coastguard Worker 130*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\intra_directional.svg" 131*77c1e3ccSAndroid Build Coastguard Workeralt="Directional intra" width="300" /> <figcaption>Figure 4: Directional intra 132*77c1e3ccSAndroid Build Coastguard Workerprediction modes</figcaption> </figure> 133*77c1e3ccSAndroid Build Coastguard Worker 134*77c1e3ccSAndroid Build Coastguard WorkerThe nominal mode index and angle delta index is signalled separately, and 135*77c1e3ccSAndroid Build Coastguard Workernominal mode index is signalled before the associated angle delta index. It is 136*77c1e3ccSAndroid Build Coastguard Workernoted that for small block sizes, where the coding gain from extending intra 137*77c1e3ccSAndroid Build Coastguard Workerprediction angles may saturate, only the nominal modes are used and angle delta 138*77c1e3ccSAndroid Build Coastguard Workerindex is not signalled. 139*77c1e3ccSAndroid Build Coastguard Worker 140*77c1e3ccSAndroid Build Coastguard Worker### Non-directional intra prediction modes 141*77c1e3ccSAndroid Build Coastguard Worker 142*77c1e3ccSAndroid Build Coastguard WorkerIn addition to directional intra prediction modes, four non-directional intra 143*77c1e3ccSAndroid Build Coastguard Workermodes which simulate smooth textures are also included. The four non-directional 144*77c1e3ccSAndroid Build Coastguard Workerintra modes include `SMOOTH_V`, `SMOOTH_H`, `SMOOTH` and `PAETH predictor`. 145*77c1e3ccSAndroid Build Coastguard Worker 146*77c1e3ccSAndroid Build Coastguard WorkerIn `SMOOTH V`, `SMOOTH H` and `SMOOTH modes`, the prediction values are 147*77c1e3ccSAndroid Build Coastguard Workergenerated using quadratic interpolation along vertical, horizontal directions, 148*77c1e3ccSAndroid Build Coastguard Workeror the average thereof. The samples used in the quadratic interpolation include 149*77c1e3ccSAndroid Build Coastguard Workerreconstructed samples from the top and left neighboring blocks and samples from 150*77c1e3ccSAndroid Build Coastguard Workerthe right and bottom boundaries which are approximated by top reconstructed 151*77c1e3ccSAndroid Build Coastguard Workersamples and the left reconstructed samples. 152*77c1e3ccSAndroid Build Coastguard Worker 153*77c1e3ccSAndroid Build Coastguard WorkerIn `PAETH predictor` mode, the prediction for each sample is assigned as one 154*77c1e3ccSAndroid Build Coastguard Workerfrom the top (T), left (L) and top-left (TL) reference samples, which has the 155*77c1e3ccSAndroid Build Coastguard Workervalue closest to the Paeth predictor value, i.e., T + L -TL. The samples used in 156*77c1e3ccSAndroid Build Coastguard Worker`PAETH predictor` are illustrated in below figure. 157*77c1e3ccSAndroid Build Coastguard Worker 158*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\intra_paeth.svg" alt="Directional 159*77c1e3ccSAndroid Build Coastguard Workerintra" width="300" /> <figcaption>Figure 5: Paeth predictor</figcaption> 160*77c1e3ccSAndroid Build Coastguard Worker</figure> 161*77c1e3ccSAndroid Build Coastguard Worker 162*77c1e3ccSAndroid Build Coastguard Worker### Recursive filtering modes 163*77c1e3ccSAndroid Build Coastguard Worker 164*77c1e3ccSAndroid Build Coastguard WorkerFive filtering intra modes are defined, and each mode specify a set of eight 165*77c1e3ccSAndroid Build Coastguard Worker7-tap filters. Given the selected filtering mode index (0~4), the current block 166*77c1e3ccSAndroid Build Coastguard Workeris divided into 4x2 sub-blocks. For one 4×2 sub-block, each sample is predicted 167*77c1e3ccSAndroid Build Coastguard Workerby 7-tap interpolation using the 7 top and left neighboring samples as inputs. 168*77c1e3ccSAndroid Build Coastguard WorkerDifferent filters are applied for samples located at different coordinates 169*77c1e3ccSAndroid Build Coastguard Workerwithin a 4×2 sub-block. The prediction process can be done recursively in unit 170*77c1e3ccSAndroid Build Coastguard Worker4x2 sub-block, which means that prediction samples generated for one 4x2 171*77c1e3ccSAndroid Build Coastguard Workerprediction block can be used to predict another 4x2 sub-block. 172*77c1e3ccSAndroid Build Coastguard Worker 173*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\intra_recursive.svg" 174*77c1e3ccSAndroid Build Coastguard Workeralt="Directional intra" width="300" /> <figcaption>Figure 6: Recursive filtering 175*77c1e3ccSAndroid Build Coastguard Workermodes</figcaption> </figure> 176*77c1e3ccSAndroid Build Coastguard Worker 177*77c1e3ccSAndroid Build Coastguard Worker### Chroma from Luma mode 178*77c1e3ccSAndroid Build Coastguard Worker 179*77c1e3ccSAndroid Build Coastguard WorkerChroma from Luma (CfL) is a chroma intra prediction mode, which models chroma 180*77c1e3ccSAndroid Build Coastguard Workersamples as a linear function of co-located reconstructed luma samples. To align 181*77c1e3ccSAndroid Build Coastguard Workerthe resolution between luma and chroma samples for different chroma sampling 182*77c1e3ccSAndroid Build Coastguard Workerformat, e.g., 4:2:0 and 4:2:2, reconstructed luma pixels may need to be 183*77c1e3ccSAndroid Build Coastguard Workersub-sampled before being used in CfL mode. In addition, the DC component is 184*77c1e3ccSAndroid Build Coastguard Workerremoved to form the AC contribution. In CfL mode, the model parameters which 185*77c1e3ccSAndroid Build Coastguard Workerspecify the linear function between two color components are optimized by 186*77c1e3ccSAndroid Build Coastguard Workerencoder signalled in the bitstream. 187*77c1e3ccSAndroid Build Coastguard Worker 188*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\intra_cfl.svg" alt="Directional 189*77c1e3ccSAndroid Build Coastguard Workerintra" width="700" /> <figcaption>Figure 7: CfL prediction</figcaption> 190*77c1e3ccSAndroid Build Coastguard Worker</figure> 191*77c1e3ccSAndroid Build Coastguard Worker 192*77c1e3ccSAndroid Build Coastguard Worker## Inter Prediction 193*77c1e3ccSAndroid Build Coastguard Worker 194*77c1e3ccSAndroid Build Coastguard Worker### Motion vector prediction 195*77c1e3ccSAndroid Build Coastguard Worker 196*77c1e3ccSAndroid Build Coastguard WorkerMotion vectors are predicted by neighboring blocks which can be either spatial 197*77c1e3ccSAndroid Build Coastguard Workerneighboring blocks, or temporal neighboring blocks located in a reference frame. 198*77c1e3ccSAndroid Build Coastguard WorkerA set of MV predictors will be identified by checking all these blocks and 199*77c1e3ccSAndroid Build Coastguard Workerutilized to encode the motion vector information. 200*77c1e3ccSAndroid Build Coastguard Worker 201*77c1e3ccSAndroid Build Coastguard Worker**Spatial motion vector prediction** 202*77c1e3ccSAndroid Build Coastguard Worker 203*77c1e3ccSAndroid Build Coastguard WorkerThere are two sets of spatial neighboring blocks that can be utilized for 204*77c1e3ccSAndroid Build Coastguard Workerfinding spatial MV predictors, including the adjacent spatial neighbors which 205*77c1e3ccSAndroid Build Coastguard Workerare direct top and left neighbors of the current block, and second outer spatial 206*77c1e3ccSAndroid Build Coastguard Workerneighbors which are close but not directly adjacent to the current block. The 207*77c1e3ccSAndroid Build Coastguard Workertwo sets of spatial neighboring blocks are illustrated in an example shown in 208*77c1e3ccSAndroid Build Coastguard WorkerFigure 8. 209*77c1e3ccSAndroid Build Coastguard Worker 210*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\inter_spatial_mvp.svg" 211*77c1e3ccSAndroid Build Coastguard Workeralt="Directional intra" width="350" /><figcaption>Figure 8: Motion field 212*77c1e3ccSAndroid Build Coastguard Workerestimation by linear projection</figcaption></figure> 213*77c1e3ccSAndroid Build Coastguard Worker 214*77c1e3ccSAndroid Build Coastguard WorkerFor each set of spatial neighbors, the top row will be checked from left to 215*77c1e3ccSAndroid Build Coastguard Workerright and then the left column will be checked from top to down. For the 216*77c1e3ccSAndroid Build Coastguard Workeradjacent spatial neighbors, an additional top-right block will be also checked 217*77c1e3ccSAndroid Build Coastguard Workerafter checking the left column neighboring blocks. For the non-adjacent spatial 218*77c1e3ccSAndroid Build Coastguard Workerneighbors, the top-left block located at (-1, -1) position will be checked 219*77c1e3ccSAndroid Build Coastguard Workerfirst, then the top row and left column in a similar manner as the adjacent 220*77c1e3ccSAndroid Build Coastguard Workerneighbors. The adjacent neighbors will be checked first, then the temporal MV 221*77c1e3ccSAndroid Build Coastguard Workerpredictor that will be described in the next subsection will be checked second, 222*77c1e3ccSAndroid Build Coastguard Workerafter that, the non-adjacent spatial neighboring blocks will be checked. 223*77c1e3ccSAndroid Build Coastguard Worker 224*77c1e3ccSAndroid Build Coastguard WorkerFor compound prediction which utilizes a pair of reference frames, the 225*77c1e3ccSAndroid Build Coastguard Workernon-adjacent spatial neighbors are not used for deriving the MV predictor. 226*77c1e3ccSAndroid Build Coastguard Worker 227*77c1e3ccSAndroid Build Coastguard Worker**Temporal motion vector prediction** 228*77c1e3ccSAndroid Build Coastguard Worker 229*77c1e3ccSAndroid Build Coastguard WorkerIn addition to spatial neighboring blocks, MV predictor can be also derived 230*77c1e3ccSAndroid Build Coastguard Workerusing co-located blocks of reference pictures, namely temporal MV predictor. To 231*77c1e3ccSAndroid Build Coastguard Workergenerate temporal MV predictor, the MVs of reference frames are first stored 232*77c1e3ccSAndroid Build Coastguard Workertogether with reference indices associated with the reference frame. Then for 233*77c1e3ccSAndroid Build Coastguard Workereach 8x8 block of the current frame, the MVs of a reference frame which pass the 234*77c1e3ccSAndroid Build Coastguard Worker8x8 block are identified and stored together with the reference frame index in a 235*77c1e3ccSAndroid Build Coastguard Workertemporal MV buffer. In an example shown in Figure 5, the MV of reference frame 1 236*77c1e3ccSAndroid Build Coastguard Worker(R1) pointing from R1 to a reference frame of R1 is identified, i.e., MVref, 237*77c1e3ccSAndroid Build Coastguard Workerwhich passes a 8x8 block (shaded in blue dots) of current frame. Then this MVref 238*77c1e3ccSAndroid Build Coastguard Workeris stored in the temporal MV buffer associated with this 8x8 block. <figure 239*77c1e3ccSAndroid Build Coastguard Workerclass="image"> <center><img src="img\inter_motion_field.svg" alt="Directional 240*77c1e3ccSAndroid Build Coastguard Workerintra" width="800" /><figcaption>Figure 9: Motion field estimation by linear 241*77c1e3ccSAndroid Build Coastguard Workerprojection</figcaption></figure> Finally, given a couple of pre-defined block 242*77c1e3ccSAndroid Build Coastguard Workercoordinates, the associated MVs stored in the temporal MV buffer are identified 243*77c1e3ccSAndroid Build Coastguard Workerand projected accordingly to derive a temporal MV predictor which points from 244*77c1e3ccSAndroid Build Coastguard Workerthe current block to its reference frame, e.g., MV0 in Figure 5. In Figure 6, 245*77c1e3ccSAndroid Build Coastguard Workerthe pre-defined block positions for deriving temporal MV predictors of a 16x16 246*77c1e3ccSAndroid Build Coastguard Workerblock are shown and up to 7 blocks will be checked to find valid temporal MV 247*77c1e3ccSAndroid Build Coastguard Workerpredictors.<figure class="image"> <center><img 248*77c1e3ccSAndroid Build Coastguard Workersrc="img\inter_tmvp_positions.svg" alt="Directional intra" width="300" 249*77c1e3ccSAndroid Build Coastguard Worker/><figcaption>Figure 10: Block positions for deriving temporal MV 250*77c1e3ccSAndroid Build Coastguard Workerpredictors</figcaption></figure> The temporal MV predictors are checked after 251*77c1e3ccSAndroid Build Coastguard Workerthe nearest spatial MV predictors but before the non-adjacent spatial MV 252*77c1e3ccSAndroid Build Coastguard Workerpredictors. 253*77c1e3ccSAndroid Build Coastguard Worker 254*77c1e3ccSAndroid Build Coastguard WorkerAll the spatial and temporal MV candidates will be put together in a pool, with 255*77c1e3ccSAndroid Build Coastguard Workereach predictor associated with a weighting determined during the scanning of the 256*77c1e3ccSAndroid Build Coastguard Workerspatial and temporal neighboring blocks. Based on the associated weightings, the 257*77c1e3ccSAndroid Build Coastguard Workercandidates are sorted and ranked, and up to four candidates will be used as a 258*77c1e3ccSAndroid Build Coastguard Workerlist MV predictor list. 259*77c1e3ccSAndroid Build Coastguard Worker 260*77c1e3ccSAndroid Build Coastguard Worker### Motion vector coding 261*77c1e3ccSAndroid Build Coastguard Worker 262*77c1e3ccSAndroid Build Coastguard Worker### Interpolation filter for motion compensation 263*77c1e3ccSAndroid Build Coastguard Worker 264*77c1e3ccSAndroid Build Coastguard Worker<mark>[Ed.: to be added]</mark> 265*77c1e3ccSAndroid Build Coastguard Worker 266*77c1e3ccSAndroid Build Coastguard Worker### Warped motion compensation 267*77c1e3ccSAndroid Build Coastguard Worker 268*77c1e3ccSAndroid Build Coastguard Worker**Global warped motion** 269*77c1e3ccSAndroid Build Coastguard Worker 270*77c1e3ccSAndroid Build Coastguard WorkerThe global motion information is signalled at each inter frame, wherein the 271*77c1e3ccSAndroid Build Coastguard Workerglobal motion type and motion parameters are included. The global motion types 272*77c1e3ccSAndroid Build Coastguard Workerand the number of the associated parameters are listed in the following table. 273*77c1e3ccSAndroid Build Coastguard Worker 274*77c1e3ccSAndroid Build Coastguard Worker 275*77c1e3ccSAndroid Build Coastguard Worker| Global motion type | Number of parameters | 276*77c1e3ccSAndroid Build Coastguard Worker|:------------------:|:--------------------:| 277*77c1e3ccSAndroid Build Coastguard Worker| Identity (zero motion)| 0 | 278*77c1e3ccSAndroid Build Coastguard Worker| Translation | 2 | 279*77c1e3ccSAndroid Build Coastguard Worker| Rotzoom | 4 | 280*77c1e3ccSAndroid Build Coastguard Worker| General affine | 6 | 281*77c1e3ccSAndroid Build Coastguard Worker 282*77c1e3ccSAndroid Build Coastguard WorkerFor an inter coded block, after the reference frame index is 283*77c1e3ccSAndroid Build Coastguard Workertransmitted, if the motion of current block is indicated as global motion, the 284*77c1e3ccSAndroid Build Coastguard Workerglobal motion type and the associated parameters of the given reference will be 285*77c1e3ccSAndroid Build Coastguard Workerused for current block. 286*77c1e3ccSAndroid Build Coastguard Worker 287*77c1e3ccSAndroid Build Coastguard Worker**Local warped motion** 288*77c1e3ccSAndroid Build Coastguard Worker 289*77c1e3ccSAndroid Build Coastguard WorkerFor an inter coded block, local warped motion is allowed when the following 290*77c1e3ccSAndroid Build Coastguard Workerconditions are all satisfied: 291*77c1e3ccSAndroid Build Coastguard Worker 292*77c1e3ccSAndroid Build Coastguard Worker* Current block is single prediction 293*77c1e3ccSAndroid Build Coastguard Worker* Width or height is greater than or equal to 8 samples 294*77c1e3ccSAndroid Build Coastguard Worker* At least one of the immediate neighbors uses same reference frame with current block 295*77c1e3ccSAndroid Build Coastguard Worker 296*77c1e3ccSAndroid Build Coastguard WorkerIf the local warped motion is used for current block, instead of signalling the 297*77c1e3ccSAndroid Build Coastguard Workeraffine parameters, they are estimated by using mean square minimization of the 298*77c1e3ccSAndroid Build Coastguard Workerdistance between the reference projection and modeled projection based on the 299*77c1e3ccSAndroid Build Coastguard Workermotion vectors of current block and its immediate neighbors. To estimate the 300*77c1e3ccSAndroid Build Coastguard Workerparameters of local warped motion, the projection sample pair of the center 301*77c1e3ccSAndroid Build Coastguard Workerpixel in neighboring block and its corresponding pixel in the reference frame 302*77c1e3ccSAndroid Build Coastguard Workerare collected if the neighboring block uses the same reference frame with 303*77c1e3ccSAndroid Build Coastguard Workercurrent block. After that, 3 extra samples are created by shifting the center 304*77c1e3ccSAndroid Build Coastguard Workerposition by a quarter sample in one or two dimensions, and these samples are 305*77c1e3ccSAndroid Build Coastguard Workeralso considered as projection sample pairs to ensure the stability of the model 306*77c1e3ccSAndroid Build Coastguard Workerparameter estimation process. 307*77c1e3ccSAndroid Build Coastguard Worker 308*77c1e3ccSAndroid Build Coastguard Worker 309*77c1e3ccSAndroid Build Coastguard Worker### Overlapped block motion compensation 310*77c1e3ccSAndroid Build Coastguard Worker 311*77c1e3ccSAndroid Build Coastguard WorkerFor an inter-coded block, overlapped block motion compensation (OBMC) is allowed 312*77c1e3ccSAndroid Build Coastguard Workerwhen the following conditions are all satisfied. 313*77c1e3ccSAndroid Build Coastguard Worker 314*77c1e3ccSAndroid Build Coastguard Worker* Current block is single prediction 315*77c1e3ccSAndroid Build Coastguard Worker* Width or height is greater than or equal to 8 samples 316*77c1e3ccSAndroid Build Coastguard Worker* At least one of the neighboring blocks are inter-coded blocks 317*77c1e3ccSAndroid Build Coastguard Worker 318*77c1e3ccSAndroid Build Coastguard WorkerWhen OBMC is applied to current block, firstly, the initial inter prediction 319*77c1e3ccSAndroid Build Coastguard Workersamples is generated by using the assigned motion vector of current block, then 320*77c1e3ccSAndroid Build Coastguard Workerthe inter predicted samples for the current block and inter predicted samples 321*77c1e3ccSAndroid Build Coastguard Workerbased on motion vectors from the above and left blocks are blended to generate 322*77c1e3ccSAndroid Build Coastguard Workerthe final prediction samples.The maximum number of neighboring motion vectors is 323*77c1e3ccSAndroid Build Coastguard Workerlimited based on the size of current block, and up to 4 motion vectors from each 324*77c1e3ccSAndroid Build Coastguard Workerof upper and left blocks can be involved in the OBMC process of current block. 325*77c1e3ccSAndroid Build Coastguard Worker 326*77c1e3ccSAndroid Build Coastguard WorkerOne example of the processing order of neighboring blocks is shown in the 327*77c1e3ccSAndroid Build Coastguard Workerfollowing picture, wherein the values marked in each block indicate the 328*77c1e3ccSAndroid Build Coastguard Workerprocessing order of the motion vectors of current block and neighboring blocks. 329*77c1e3ccSAndroid Build Coastguard WorkerTo be specific, the motion vector of current block is firstly applied to 330*77c1e3ccSAndroid Build Coastguard Workergenerate inter prediction samples P0(x,y). Then motion vector of block 1 is 331*77c1e3ccSAndroid Build Coastguard Workerapplied to generate the prediction samples p1(x,y). After that, the prediction 332*77c1e3ccSAndroid Build Coastguard Workersamples in the overlapping area between block 0 and block 1 is an weighted 333*77c1e3ccSAndroid Build Coastguard Workeraverage of p0(x,y) and p1(x,y). The overlapping area of block 1 and block 0 is 334*77c1e3ccSAndroid Build Coastguard Workermarked in grey in the following picture. The motion vectors of block 2, 3, 4 are 335*77c1e3ccSAndroid Build Coastguard Workerfurther applied and blended in the same way. 336*77c1e3ccSAndroid Build Coastguard Worker 337*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\inter_obmc.svg" alt="Directional 338*77c1e3ccSAndroid Build Coastguard Workerintra" width="300" /><figcaption>Figure 11: neighboring blocks for OBMC 339*77c1e3ccSAndroid Build Coastguard Workerprocess</figcaption></figure> 340*77c1e3ccSAndroid Build Coastguard Worker 341*77c1e3ccSAndroid Build Coastguard Worker### Reference frames 342*77c1e3ccSAndroid Build Coastguard Worker 343*77c1e3ccSAndroid Build Coastguard Worker<mark>[Ed.: to be added]</mark> 344*77c1e3ccSAndroid Build Coastguard Worker 345*77c1e3ccSAndroid Build Coastguard Worker### Compound Prediction 346*77c1e3ccSAndroid Build Coastguard Worker 347*77c1e3ccSAndroid Build Coastguard Worker<mark>[Ed.: to be added]</mark> 348*77c1e3ccSAndroid Build Coastguard Worker 349*77c1e3ccSAndroid Build Coastguard Worker**Compound wedge prediction** 350*77c1e3ccSAndroid Build Coastguard Worker 351*77c1e3ccSAndroid Build Coastguard Worker<mark>[Ed.: to be added]</mark> 352*77c1e3ccSAndroid Build Coastguard Worker 353*77c1e3ccSAndroid Build Coastguard Worker**Difference-modulated masked prediction** 354*77c1e3ccSAndroid Build Coastguard Worker 355*77c1e3ccSAndroid Build Coastguard Worker<mark>[Ed.: to be added]</mark> 356*77c1e3ccSAndroid Build Coastguard Worker 357*77c1e3ccSAndroid Build Coastguard Worker**Frame distance-based compound prediction** 358*77c1e3ccSAndroid Build Coastguard Worker 359*77c1e3ccSAndroid Build Coastguard Worker<mark>[Ed.: to be added]</mark> 360*77c1e3ccSAndroid Build Coastguard Worker 361*77c1e3ccSAndroid Build Coastguard Worker**Compound inter-intra prediction** 362*77c1e3ccSAndroid Build Coastguard Worker 363*77c1e3ccSAndroid Build Coastguard Worker<mark>[Ed.: to be added]</mark> 364*77c1e3ccSAndroid Build Coastguard Worker 365*77c1e3ccSAndroid Build Coastguard Worker## Transform 366*77c1e3ccSAndroid Build Coastguard Worker 367*77c1e3ccSAndroid Build Coastguard WorkerThe separable 2D transform process is applied on prediction residuals. For the 368*77c1e3ccSAndroid Build Coastguard Workerforward transform, a 1-D vertical transform is performed first on each column of 369*77c1e3ccSAndroid Build Coastguard Workerthe input residual block, then a horizontal transform is performed on each row 370*77c1e3ccSAndroid Build Coastguard Workerof the vertical transform output. For the backward transform, a 1-D horizontal 371*77c1e3ccSAndroid Build Coastguard Workertransform is performed first on each row of the input de-quantized coefficient 372*77c1e3ccSAndroid Build Coastguard Workerblock, then a vertical transform is performed on each column of the horizontal 373*77c1e3ccSAndroid Build Coastguard Workertransform output. The primary 1-D transforms include four different types of 374*77c1e3ccSAndroid Build Coastguard Workertransform: a) 4-point, 8-point, 16-point, 32-point, 64-point DCT-2; b) 4-point, 375*77c1e3ccSAndroid Build Coastguard Worker8-point, 16-point asymmetric DST’s (DST-4, DST-7) and c) their flipped 376*77c1e3ccSAndroid Build Coastguard Workerversions; d) 4-point, 8-point, 16-point, 32-point identity transforms. When 377*77c1e3ccSAndroid Build Coastguard Workertransform size is 4-point, ADST refers to DST-7, otherwise, when transform size 378*77c1e3ccSAndroid Build Coastguard Workeris greater than 4-point, ADST refers to DST-4. 379*77c1e3ccSAndroid Build Coastguard Worker 380*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><figcaption>Table 2: Transform basis functions 381*77c1e3ccSAndroid Build Coastguard Worker(DCT-2, DST-4 and DST-7 for N-point input.</figcaption> <img src= 382*77c1e3ccSAndroid Build Coastguard Worker"img\tx_basis.svg" alt="Partition" width="450" /> </figure> 383*77c1e3ccSAndroid Build Coastguard Worker 384*77c1e3ccSAndroid Build Coastguard WorkerFor luma component, each transform block can select one pair of horizontal and 385*77c1e3ccSAndroid Build Coastguard Workervertical transform combination given a pre-defined set of transform type 386*77c1e3ccSAndroid Build Coastguard Workercandidates, and the selection is explicitly signalled into the bitstream. 387*77c1e3ccSAndroid Build Coastguard WorkerHowever, the selection is not signalled when Max(width,height) is 64. When 388*77c1e3ccSAndroid Build Coastguard Workerthe maximum of transform block width and height is greater than or equal to 32, 389*77c1e3ccSAndroid Build Coastguard Workerthe set of transform type candidates depend on the prediction mode, as described 390*77c1e3ccSAndroid Build Coastguard Workerin Table 3. Otherwise, when the maximum of transform block width and height is 391*77c1e3ccSAndroid Build Coastguard Workersmaller than 32, the set of transform type candidates depend on the prediction 392*77c1e3ccSAndroid Build Coastguard Workermode, as described in Table 4. 393*77c1e3ccSAndroid Build Coastguard Worker 394*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><figcaption>Table 3: Transform type candidates 395*77c1e3ccSAndroid Build Coastguard Workerfor luma component when max(width, height) is greater than or equal to 32. 396*77c1e3ccSAndroid Build Coastguard Worker</figcaption> <img src="img\tx_cands_large.svg" alt="Partition" width="370" /> 397*77c1e3ccSAndroid Build Coastguard Worker</figure> 398*77c1e3ccSAndroid Build Coastguard Worker 399*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><figcaption>Table 4: Transform type candidates 400*77c1e3ccSAndroid Build Coastguard Workerfor luma component when max(width, height) is smaller than 32. </figcaption> 401*77c1e3ccSAndroid Build Coastguard Worker<img src="img\tx_cands_small.svg" alt="Partition" width="440" /> </figure> 402*77c1e3ccSAndroid Build Coastguard Worker 403*77c1e3ccSAndroid Build Coastguard WorkerThe set of transform type candidates (namely transform set) is defined in Table 404*77c1e3ccSAndroid Build Coastguard Worker5. 405*77c1e3ccSAndroid Build Coastguard Worker 406*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><figcaption>Table 5: Definition of transform set. 407*77c1e3ccSAndroid Build Coastguard Worker</figcaption> <img src="img\tx_set.svg" alt="Partition" width="450" /> </figure> 408*77c1e3ccSAndroid Build Coastguard Worker 409*77c1e3ccSAndroid Build Coastguard WorkerFor chroma component, the transform type selection is done in an implicit way. 410*77c1e3ccSAndroid Build Coastguard WorkerFor intra prediction residuals, the transform type is selected according to the 411*77c1e3ccSAndroid Build Coastguard Workerintra prediction mode, as specified in Table 4. For inter prediction residuals, 412*77c1e3ccSAndroid Build Coastguard Workerthe transform type is selected according to the transform type selection of the 413*77c1e3ccSAndroid Build Coastguard Workerco-located luma block. Therefore, for chroma component, there is no transform 414*77c1e3ccSAndroid Build Coastguard Workertype signalling in the bitstream. 415*77c1e3ccSAndroid Build Coastguard Worker 416*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><figcaption>Table 6: Transform type selection for 417*77c1e3ccSAndroid Build Coastguard Workerchroma component intra prediction residuals.</figcaption> <img src= 418*77c1e3ccSAndroid Build Coastguard Worker"img\tx_chroma.svg" alt="Partition" width="500" /> </figure> 419*77c1e3ccSAndroid Build Coastguard Worker 420*77c1e3ccSAndroid Build Coastguard WorkerThe computational cost of large size (e.g., 64-point) transforms is further 421*77c1e3ccSAndroid Build Coastguard Workerreduced by zeroing out all the coefficients except the following two cases: 422*77c1e3ccSAndroid Build Coastguard Worker 423*77c1e3ccSAndroid Build Coastguard Worker1. The top-left 32×32 quadrant for 64×64/64×32/32×64 DCT_DCT hybrid transforms 424*77c1e3ccSAndroid Build Coastguard Worker2. The left 32×16 area for 64×16 and top 16×32 for16×64 DCT_DCT hybrid transforms. 425*77c1e3ccSAndroid Build Coastguard Worker 426*77c1e3ccSAndroid Build Coastguard WorkerBoth the DCT-2 and ADST (DST-4, DST-7) are implemented using butterfly structure 427*77c1e3ccSAndroid Build Coastguard Worker[1], which included multiple stages of butterfly operations. Each butterfly 428*77c1e3ccSAndroid Build Coastguard Workeroperations can be calculated in parallel and different stages are cascaded in a 429*77c1e3ccSAndroid Build Coastguard Workersequential order. 430*77c1e3ccSAndroid Build Coastguard Worker 431*77c1e3ccSAndroid Build Coastguard Worker## Quantization 432*77c1e3ccSAndroid Build Coastguard WorkerQuantization of transform coefficients may apply different quantization step 433*77c1e3ccSAndroid Build Coastguard Workersize for DC and AC transform coefficients, and different quantization step size 434*77c1e3ccSAndroid Build Coastguard Workerfor luma and chroma transform coefficients. To specify the quantization step 435*77c1e3ccSAndroid Build Coastguard Workersize, in the frame header, a _**base_q_idx**_ syntax element is first signalled, 436*77c1e3ccSAndroid Build Coastguard Workerwhich is a 8-bit fixed length code specifying the quantization step size for 437*77c1e3ccSAndroid Build Coastguard Workerluma AC coefficients. The valid range of _**base_q_idx**_ is [0, 255]. 438*77c1e3ccSAndroid Build Coastguard Worker 439*77c1e3ccSAndroid Build Coastguard WorkerAfter that, the delta value relative to base_q_idx for Luma DC coefficients, 440*77c1e3ccSAndroid Build Coastguard Workerindicated as DeltaQYDc is further signalled. Furthermore, if there are more than 441*77c1e3ccSAndroid Build Coastguard Workerone color plane, then a flag _**diff_uv_delta**_ is signaled to indicate whether 442*77c1e3ccSAndroid Build Coastguard WorkerCb and Cr color components apply different quantization index values. If 443*77c1e3ccSAndroid Build Coastguard Worker_**diff_uv_delta**_ is signalled as 0, then only the delta values relative to 444*77c1e3ccSAndroid Build Coastguard Workerbase_q_idx for chroma DC coefficients (indicated as DeltaQUDc) and AC 445*77c1e3ccSAndroid Build Coastguard Workercoefficients (indicated as DeltaQUAc) are signalled. Otherwise, the delta values 446*77c1e3ccSAndroid Build Coastguard Workerrelative to base_q_idx for both the Cb and Cr DC coefficients (indicated as 447*77c1e3ccSAndroid Build Coastguard WorkerDeltaQUDc and DeltaQVDc) and AC coefficients (indicated as DeltaQUAc and 448*77c1e3ccSAndroid Build Coastguard WorkerDeltaQVAc) are signalled. 449*77c1e3ccSAndroid Build Coastguard Worker 450*77c1e3ccSAndroid Build Coastguard WorkerThe above decoded DeltaQYDc, DeltaQUAc, DeltaQUDc, DeltaQVAc and DeltaQVDc are 451*77c1e3ccSAndroid Build Coastguard Workeradded to _base_q_idx_ to derive the quantization indices. Then these 452*77c1e3ccSAndroid Build Coastguard Workerquantization indices are further mapped to quantization step size according to 453*77c1e3ccSAndroid Build Coastguard Workertwo tables. For DC coefficients, the mapping from quantization index to 454*77c1e3ccSAndroid Build Coastguard Workerquantization step size for 8-bit, 10-bit and 12-bit internal bit depth is 455*77c1e3ccSAndroid Build Coastguard Workerspecified by a lookup table Dc_Qlookup[3][256], and the mapping from 456*77c1e3ccSAndroid Build Coastguard Workerquantization index to quantization step size for 8-bit, 10-bit and 12-bit is 457*77c1e3ccSAndroid Build Coastguard Workerspecified by a lookup table Ac_Qlookup[3][256]. 458*77c1e3ccSAndroid Build Coastguard Worker 459*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\quant_dc.svg" alt="quant_dc" 460*77c1e3ccSAndroid Build Coastguard Workerwidth="800" /><figcaption>Figure 11: Quantization step size of DC coefficients 461*77c1e3ccSAndroid Build Coastguard Workerfor different internal bit-depth</figcaption></figure> 462*77c1e3ccSAndroid Build Coastguard Worker 463*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\quant_ac.svg" alt="quant_ac" 464*77c1e3ccSAndroid Build Coastguard Workerwidth="800" /><figcaption>Figure 12: Quantization step size of AC coefficients 465*77c1e3ccSAndroid Build Coastguard Workerfor different internal bit-depth</figcaption></figure> 466*77c1e3ccSAndroid Build Coastguard Worker 467*77c1e3ccSAndroid Build Coastguard WorkerGiven the quantization step size, indicated as _Q<sub>step_, the input quantized 468*77c1e3ccSAndroid Build Coastguard Workercoefficients is further de-quantized using the following formula: 469*77c1e3ccSAndroid Build Coastguard Worker 470*77c1e3ccSAndroid Build Coastguard Worker_F_ = sign * ( (_f_ * _Q<sub>step_) % 0xFFFFFF ) / _deNorm_ 471*77c1e3ccSAndroid Build Coastguard Worker 472*77c1e3ccSAndroid Build Coastguard Worker, where _f_ is the input quantized coefficient, _F_ is the output dequantized 473*77c1e3ccSAndroid Build Coastguard Workercoefficient, _deNorm_ is a constant value derived from the transform block area 474*77c1e3ccSAndroid Build Coastguard Workersize, as indicated by the following table: 475*77c1e3ccSAndroid Build Coastguard Worker 476*77c1e3ccSAndroid Build Coastguard Worker| _deNorm_ | Tx block area size | 477*77c1e3ccSAndroid Build Coastguard Worker|----------|:--------------------------| 478*77c1e3ccSAndroid Build Coastguard Worker| 1| Less than 512 samples | 479*77c1e3ccSAndroid Build Coastguard Worker| 2 | 512 or 1024 samples | 480*77c1e3ccSAndroid Build Coastguard Worker| 4 | Greater than 1024 samples | 481*77c1e3ccSAndroid Build Coastguard Worker 482*77c1e3ccSAndroid Build Coastguard WorkerWhen the quantization index is 0, the quantization is performed using a 483*77c1e3ccSAndroid Build Coastguard Workerquantization step size equal to 1, which is lossless coding mode. 484*77c1e3ccSAndroid Build Coastguard Worker 485*77c1e3ccSAndroid Build Coastguard Worker## Entropy Coding 486*77c1e3ccSAndroid Build Coastguard Worker 487*77c1e3ccSAndroid Build Coastguard Worker**Entropy coding engine** 488*77c1e3ccSAndroid Build Coastguard Worker 489*77c1e3ccSAndroid Build Coastguard Worker<mark>[Ed.: to be added]</mark> 490*77c1e3ccSAndroid Build Coastguard Worker 491*77c1e3ccSAndroid Build Coastguard Worker**Coefficient coding** 492*77c1e3ccSAndroid Build Coastguard Worker 493*77c1e3ccSAndroid Build Coastguard WorkerFor each transform unit, the coefficient coding starts with coding a skip sign, 494*77c1e3ccSAndroid Build Coastguard Workerwhich is followed by the signaling of primary transform kernel type and the 495*77c1e3ccSAndroid Build Coastguard Workerend-of-block (EOB) position in case the transform coding is not skipped. After 496*77c1e3ccSAndroid Build Coastguard Workerthat, the coefficient values are coded in a multiple level map manner plus sign 497*77c1e3ccSAndroid Build Coastguard Workervalues. The level maps are coded as three level planes, namely lower-level, 498*77c1e3ccSAndroid Build Coastguard Workermiddle-level and higher-level planes, and the sign is coded as another separate 499*77c1e3ccSAndroid Build Coastguard Workerplane. The lower-level, middle-level and higher-level planes correspond to 500*77c1e3ccSAndroid Build Coastguard Workercorrespond to different ranges of coefficient magnitudes. The lower level plane 501*77c1e3ccSAndroid Build Coastguard Workercorresponds to the range of 0–2, the middle level plane takes care of the 502*77c1e3ccSAndroid Build Coastguard Workerrange of 3–14, and the higher-level plane covers the range of 15 and above. 503*77c1e3ccSAndroid Build Coastguard Worker 504*77c1e3ccSAndroid Build Coastguard WorkerThe three level planes are coded as follows. After the EOB position is coded, 505*77c1e3ccSAndroid Build Coastguard Workerthe lower-level and middle-level planes are coded together in backward scan 506*77c1e3ccSAndroid Build Coastguard Workerorder, and the scan order refers to zig-zag scan applied on the entire transform 507*77c1e3ccSAndroid Build Coastguard Workerunit basis. Then the sign plane and higher-level plane are coded together in 508*77c1e3ccSAndroid Build Coastguard Workerforward scan order. After that, the remainder (coefficient level minus 14) is 509*77c1e3ccSAndroid Build Coastguard Workerentropy coded using Exp-Golomb code. 510*77c1e3ccSAndroid Build Coastguard Worker 511*77c1e3ccSAndroid Build Coastguard WorkerThe context model applied to the lower level plane depends on the primary 512*77c1e3ccSAndroid Build Coastguard Workertransform directions, including: bi-directional, horizontal, and vertical, as 513*77c1e3ccSAndroid Build Coastguard Workerwell as transform size, and up to five neighbor (in frequency domain) 514*77c1e3ccSAndroid Build Coastguard Workercoefficients are used to derive the context. The middle level plane uses a 515*77c1e3ccSAndroid Build Coastguard Workersimilar context model, but the number of context neighbor coefficients is 516*77c1e3ccSAndroid Build Coastguard Workerreduced from 5 to 2. The higher-level plane is coded by Exp-Golomb code without 517*77c1e3ccSAndroid Build Coastguard Workerusing context model. For the sign plane, except the DC sign that is coded using 518*77c1e3ccSAndroid Build Coastguard Workerthe DC signs from its neighboring transform units, sign values of other 519*77c1e3ccSAndroid Build Coastguard Workercoefficients are coded directly without using context model. 520*77c1e3ccSAndroid Build Coastguard Worker 521*77c1e3ccSAndroid Build Coastguard Worker## Loop filtering and post-processing 522*77c1e3ccSAndroid Build Coastguard Worker 523*77c1e3ccSAndroid Build Coastguard Worker### Deblocking 524*77c1e3ccSAndroid Build Coastguard Worker 525*77c1e3ccSAndroid Build Coastguard WorkerThere are four methods when picking deblocking filter level, which are listed 526*77c1e3ccSAndroid Build Coastguard Workerbelow: 527*77c1e3ccSAndroid Build Coastguard Worker 528*77c1e3ccSAndroid Build Coastguard Worker* LPF_PICK_FROM_FULL_IMAGE: search the full image with different values 529*77c1e3ccSAndroid Build Coastguard Worker* LPF_PICK_FROM_Q: estimate the filter level based on quantizer and frame type 530*77c1e3ccSAndroid Build Coastguard Worker* LPF_PICK_FROM_SUBIMAGE: estimate the level from a portion of image 531*77c1e3ccSAndroid Build Coastguard Worker* LPF_PICK_MINIMAL_LPF: set the filter level to 0 and disable the deblocking 532*77c1e3ccSAndroid Build Coastguard Worker 533*77c1e3ccSAndroid Build Coastguard WorkerWhen estimating the filter level from the full image or sub-image, the searching 534*77c1e3ccSAndroid Build Coastguard Workerstarts from the previous frame filter level, ends when the filter step is less 535*77c1e3ccSAndroid Build Coastguard Workeror equal to zero. In addition to filter level, there are some other parameters 536*77c1e3ccSAndroid Build Coastguard Workerwhich control the deblocking filter such as sharpness level, mode deltas, and 537*77c1e3ccSAndroid Build Coastguard Workerreference deltas. 538*77c1e3ccSAndroid Build Coastguard Worker 539*77c1e3ccSAndroid Build Coastguard WorkerDeblocking is performed at 128x128 super block level, and the vertical and 540*77c1e3ccSAndroid Build Coastguard Workerhorizontal edges are filtered respectively. For a 128x128 super block, the 541*77c1e3ccSAndroid Build Coastguard Workervertical/horizontal edges aligned with each 8x8 block is firstly filtered. If 542*77c1e3ccSAndroid Build Coastguard Workerthe 4x4 transform is used, the internal edge aligned with a 4x4 block will be 543*77c1e3ccSAndroid Build Coastguard Workerfurther filtered. The filter length is switchable from 4-tap, 6-tap, 8-tap, 544*77c1e3ccSAndroid Build Coastguard Worker14-tap, and 0-tap (no filtering). The location of filter taps are identified 545*77c1e3ccSAndroid Build Coastguard Workerbased on the number of filter taps in order to compute the filter mask. When 546*77c1e3ccSAndroid Build Coastguard Workerfinally performing the filtering, outer taps are added if there is high edge 547*77c1e3ccSAndroid Build Coastguard Workervariance. 548*77c1e3ccSAndroid Build Coastguard Worker 549*77c1e3ccSAndroid Build Coastguard Worker### Constrained directional enhancement filter 550*77c1e3ccSAndroid Build Coastguard Worker 551*77c1e3ccSAndroid Build Coastguard Worker**Edge Direction Estimation**\ 552*77c1e3ccSAndroid Build Coastguard WorkerIn CDEF, edge direction search is performed at 8x8 block-level. There are 553*77c1e3ccSAndroid Build Coastguard Workereight edge directions in total, as illustrated in Figure 13. 554*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\edge_direction.svg" 555*77c1e3ccSAndroid Build Coastguard Workeralt="Edge direction" width="700" /> <figcaption>Figure 13: Line number 556*77c1e3ccSAndroid Build Coastguard Workerk for pixels following direction d=0:7 in an 8x8 block.</figcaption> </figure> 557*77c1e3ccSAndroid Build Coastguard Worker 558*77c1e3ccSAndroid Build Coastguard WorkerThe optimal edge direction d_opt is found by maximizing the following 559*77c1e3ccSAndroid Build Coastguard Workerterm [3]: 560*77c1e3ccSAndroid Build Coastguard Worker 561*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\equ_edge_direction.svg" 562*77c1e3ccSAndroid Build Coastguard Workeralt="Equation edge direction" width="250" /> </figure> 563*77c1e3ccSAndroid Build Coastguard Worker<!-- $$d_{opt}=\max_{d} s_d$$ 564*77c1e3ccSAndroid Build Coastguard Worker$$s_d = \sum_{k}\frac{1}{N_{d,k}}(\sum_{p\in P_{d,k}}x_p)^2,$$ --> 565*77c1e3ccSAndroid Build Coastguard Worker 566*77c1e3ccSAndroid Build Coastguard Workerwhere x_p is the value of pixel p, P_{d,k} is the set of pixels in 567*77c1e3ccSAndroid Build Coastguard Workerline k following direction d, N_{d,k} is the cardinality of P_{d,k}. 568*77c1e3ccSAndroid Build Coastguard Worker 569*77c1e3ccSAndroid Build Coastguard Worker**Directional filter**\ 570*77c1e3ccSAndroid Build Coastguard WorkerCDEF consists two filter taps: the primary tap and the secondary tap. 571*77c1e3ccSAndroid Build Coastguard WorkerThe primary tap works along the edge direction (as shown in Figure 14), 572*77c1e3ccSAndroid Build Coastguard Workerwhile the secondary tap forms an oriented 45 degree off the edge direction 573*77c1e3ccSAndroid Build Coastguard Worker (as shown in Figure 15). 574*77c1e3ccSAndroid Build Coastguard Worker 575*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\primary_tap.svg" 576*77c1e3ccSAndroid Build Coastguard Workeralt="Primary tap" width="700" /> <figcaption>Figure 14: Primary filter 577*77c1e3ccSAndroid Build Coastguard Workertaps following edge direction. For even strengths a = 2 and b = 4, for 578*77c1e3ccSAndroid Build Coastguard Workerodd strengths a = 3 and b = 3. The filtered pixel is shown in the 579*77c1e3ccSAndroid Build Coastguard Workerhighlighted center.</figcaption> </figure> 580*77c1e3ccSAndroid Build Coastguard Worker 581*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\secondary_tap.svg" 582*77c1e3ccSAndroid Build Coastguard Workeralt="Edge direction" width="700" /> <figcaption>Figure 15: Secondary 583*77c1e3ccSAndroid Build Coastguard Workerfilter taps. The filtered pixel is shown in the highlighted center. 584*77c1e3ccSAndroid Build Coastguard Worker</figcaption> </figure> 585*77c1e3ccSAndroid Build Coastguard Worker 586*77c1e3ccSAndroid Build Coastguard WorkerCDEF can be described by the following equation: 587*77c1e3ccSAndroid Build Coastguard Worker 588*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\equ_dir_search.svg" 589*77c1e3ccSAndroid Build Coastguard Workeralt="Equation direction search" width="720" /> </figure> 590*77c1e3ccSAndroid Build Coastguard Worker 591*77c1e3ccSAndroid Build Coastguard Worker<!-- $$y(i,j)=x(i,j)+round(\sum_{m,n}w^{(p)}_{d,m,n}f(x(m,x)-x(i,j),S^{(p)}, 592*77c1e3ccSAndroid Build Coastguard WorkerD)+\sum_{m,n}w^{(s)}_{d,m,n}f(x(m,x)-x(i,j),S^{(s)},D)),$$ --> 593*77c1e3ccSAndroid Build Coastguard Worker 594*77c1e3ccSAndroid Build Coastguard Workerwhere x(i,j) and y(i,j) are the input and output reconstructed values 595*77c1e3ccSAndroid Build Coastguard Workerof CDEF. p denotes primary tap, and s denotes secondary tap, w is 596*77c1e3ccSAndroid Build Coastguard Workerthe weight between primary and secondary tap. f(d,S,D) is a non-linear 597*77c1e3ccSAndroid Build Coastguard Workerfiltering function, S denotes filter strength, D is a damping parameter. 598*77c1e3ccSAndroid Build Coastguard WorkerFor 8-bit content, S^p ranges from 0 to 15, and S^s can be 599*77c1e3ccSAndroid Build Coastguard Worker0, 1, 2, or 4. D ranges from 3 to 6 for luma, and 2 to 4 for chroma. 600*77c1e3ccSAndroid Build Coastguard Worker 601*77c1e3ccSAndroid Build Coastguard Worker**Non linear filter**\ 602*77c1e3ccSAndroid Build Coastguard WorkerCDEF uses a non-linear filtering function to prevent excessive blurring 603*77c1e3ccSAndroid Build Coastguard Workerwhen applied across an edge. It is achieved by ignoring pixels that are 604*77c1e3ccSAndroid Build Coastguard Workertoo different from the current pixels to be filtered. When the difference 605*77c1e3ccSAndroid Build Coastguard Workerbetween current pixel and it's neighboring pixel d is within a threshold, 606*77c1e3ccSAndroid Build Coastguard Workerf(d,S,D) = d, otherwise f(d,S,D) = 0. Specifically, the strength S 607*77c1e3ccSAndroid Build Coastguard Workerdetermines the maximum difference allowed and damping D determines the 608*77c1e3ccSAndroid Build Coastguard Workerpoint to ignore the filter tap. 609*77c1e3ccSAndroid Build Coastguard Worker 610*77c1e3ccSAndroid Build Coastguard Worker### Loop Restoration filter 611*77c1e3ccSAndroid Build Coastguard Worker 612*77c1e3ccSAndroid Build Coastguard Worker**Separable symmetric wiener filter** 613*77c1e3ccSAndroid Build Coastguard Worker 614*77c1e3ccSAndroid Build Coastguard WorkerLet F be a w x w 2D filter taps around the pixel to be filtered, denoted as 615*77c1e3ccSAndroid Build Coastguard Workera w^2 x 1 column vector. When compared with traditional Wiener Filter, 616*77c1e3ccSAndroid Build Coastguard WorkerSeparable Symmetric Wiener Filter has the following three constraints in order 617*77c1e3ccSAndroid Build Coastguard Workerto save signaling bits and reduce complexity [4]: 618*77c1e3ccSAndroid Build Coastguard Worker 619*77c1e3ccSAndroid Build Coastguard Worker1) The w x w filter window of is separated into horizontal and vertical w-tap 620*77c1e3ccSAndroid Build Coastguard Workerconvolutions. 621*77c1e3ccSAndroid Build Coastguard Worker 622*77c1e3ccSAndroid Build Coastguard Worker2) The horizontal and vertical filters are constrained to be symmetric. 623*77c1e3ccSAndroid Build Coastguard Worker 624*77c1e3ccSAndroid Build Coastguard Worker3) It is assumed that the summation of horizontal/vertical filter coefficients 625*77c1e3ccSAndroid Build Coastguard Workeris 1. 626*77c1e3ccSAndroid Build Coastguard Worker 627*77c1e3ccSAndroid Build Coastguard WorkerAs a result, F can be written as F = column_vectorize[ab^T], subject to a(i) 628*77c1e3ccSAndroid Build Coastguard Worker= a(w - 1 - i), b(i) = b(w - 1 - i), for i = [0, r - 1], and sum(a(i)) = 629*77c1e3ccSAndroid Build Coastguard Workersum(b(i)) = 1, where a is the vertical filters and b is the horizontal filters. 630*77c1e3ccSAndroid Build Coastguard WorkerThe derivation of the filters a and b starts from an initial guess of 631*77c1e3ccSAndroid Build Coastguard Workerhorizontal and vertical filters, optimizing one of the two while holding the 632*77c1e3ccSAndroid Build Coastguard Workerother fixed. In the implementation w = 7, thus, 3 taps need to be sent for 633*77c1e3ccSAndroid Build Coastguard Workerfilters a and b, respectively. When signaling the filter coefficients, 4, 5 and 634*77c1e3ccSAndroid Build Coastguard Worker6 bits are used for the first three filter taps, and the remaining ones are 635*77c1e3ccSAndroid Build Coastguard Workerobtained from the normalization and symmetry constraints. 30 bits in total are 636*77c1e3ccSAndroid Build Coastguard Workertransmitted for both vertical and horizontal filters. 637*77c1e3ccSAndroid Build Coastguard Worker 638*77c1e3ccSAndroid Build Coastguard Worker 639*77c1e3ccSAndroid Build Coastguard Worker**Dual self-guided filter** 640*77c1e3ccSAndroid Build Coastguard Worker 641*77c1e3ccSAndroid Build Coastguard WorkerDual self-guided filter is designed to firstly obtain two coarse restorations 642*77c1e3ccSAndroid Build Coastguard WorkerX1 and X2 of the degraded frame X, and the final restoration Xr is obtained as 643*77c1e3ccSAndroid Build Coastguard Workera combination of the degraded samples, and the difference between the degraded 644*77c1e3ccSAndroid Build Coastguard Workersamples and the coarse restorations [4]: 645*77c1e3ccSAndroid Build Coastguard Worker 646*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\equ_dual_self_guided.svg" 647*77c1e3ccSAndroid Build Coastguard Workeralt="Equation dual self guided filter" width="300" /> </figure> 648*77c1e3ccSAndroid Build Coastguard Worker<!-- $$X_r = X + \alpha (X_1 - X) + \beta (X_2 - X)$$ --> 649*77c1e3ccSAndroid Build Coastguard Worker 650*77c1e3ccSAndroid Build Coastguard WorkerAt encoder side, alpha and beta are computed using: 651*77c1e3ccSAndroid Build Coastguard Worker 652*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\equ_dual_self_para.svg" 653*77c1e3ccSAndroid Build Coastguard Workeralt="Equation dual self guided filter parameter" width="220" /> </figure> 654*77c1e3ccSAndroid Build Coastguard Worker<!-- $${\alpha, \beta}^T = (A^T A) ^{-1} A^T b,$$ --> 655*77c1e3ccSAndroid Build Coastguard Worker 656*77c1e3ccSAndroid Build Coastguard Workerwhere A = {X1 - X, X2 - X}, b = Y - X, and Y is the original source. 657*77c1e3ccSAndroid Build Coastguard Worker 658*77c1e3ccSAndroid Build Coastguard WorkerX1 and X2 are obtained using guided filtering, and the filtering is controlled 659*77c1e3ccSAndroid Build Coastguard Workerby a radius r and a noise parameter e, where a higher r implies a higher 660*77c1e3ccSAndroid Build Coastguard Workerspatial variance and a higher e implies a higher range variance [4]. X1 and X2 661*77c1e3ccSAndroid Build Coastguard Workercan be described by {r1, e1} and {r2, e2}, respectively. 662*77c1e3ccSAndroid Build Coastguard Worker 663*77c1e3ccSAndroid Build Coastguard WorkerThe encoder sends a 6-tuple {r1, e1, r2, e2, alpha, beta} to the decoder. In 664*77c1e3ccSAndroid Build Coastguard Workerthe implementation, {r1, e1, r2, e2} uses a 3-bit codebook, and {alpha, beta} 665*77c1e3ccSAndroid Build Coastguard Workeruses 7-bit each due to much higher precision, resulting in a total of 17 bits. 666*77c1e3ccSAndroid Build Coastguard Workerr is always less or equal to 3 [4]. 667*77c1e3ccSAndroid Build Coastguard Worker 668*77c1e3ccSAndroid Build Coastguard WorkerGuided filtering can be described by a local linear model: 669*77c1e3ccSAndroid Build Coastguard Worker 670*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\equ_guided_filter.svg" 671*77c1e3ccSAndroid Build Coastguard Workeralt="Equation guided filter" width="155" /> </figure> 672*77c1e3ccSAndroid Build Coastguard Worker<!-- $$y=Fx+G,$$ --> 673*77c1e3ccSAndroid Build Coastguard Worker 674*77c1e3ccSAndroid Build Coastguard Workerwhere x and y are the input and output samples, F and G are determined by the 675*77c1e3ccSAndroid Build Coastguard Workerstatistics in the neighboring of the pixel to be filtered. It is called 676*77c1e3ccSAndroid Build Coastguard Workerself-guided filtering when the guidance image is the same as the degraded 677*77c1e3ccSAndroid Build Coastguard Workerimage[4]. 678*77c1e3ccSAndroid Build Coastguard Worker 679*77c1e3ccSAndroid Build Coastguard WorkerFollowing are three steps when deriving F and G of the self-guided filtering: 680*77c1e3ccSAndroid Build Coastguard Worker 681*77c1e3ccSAndroid Build Coastguard Worker1) Compute mean u and variance d of pixels in a (2r + 1) x (2r + 1) window 682*77c1e3ccSAndroid Build Coastguard Workeraround the pixel to be filtered. 683*77c1e3ccSAndroid Build Coastguard Worker 684*77c1e3ccSAndroid Build Coastguard Worker2) For each pixel, compute f = d / (d + e); g = (1 - f)u. 685*77c1e3ccSAndroid Build Coastguard Worker 686*77c1e3ccSAndroid Build Coastguard Worker3) Compute F and G for each pixel as averages of f and g values in a 3 x 3 687*77c1e3ccSAndroid Build Coastguard Workerwindow around the pixel for use in step 2. 688*77c1e3ccSAndroid Build Coastguard Worker 689*77c1e3ccSAndroid Build Coastguard Worker### Frame super-resolution 690*77c1e3ccSAndroid Build Coastguard Worker 691*77c1e3ccSAndroid Build Coastguard WorkerIn order to improve the perceptual quality of decoded pictures, a 692*77c1e3ccSAndroid Build Coastguard Workersuper-resolution process is applied at low bit-rates [5]. First, at encoder 693*77c1e3ccSAndroid Build Coastguard Workerside, the source video is downscaled as a non-normative procedure. Second, 694*77c1e3ccSAndroid Build Coastguard Workerthe downscaled video is encoded, followed by deblocking and CDEF process. 695*77c1e3ccSAndroid Build Coastguard WorkerThird, a linear upscaling process is applied as a normative procedure to bring 696*77c1e3ccSAndroid Build Coastguard Workerthe encoded video back to it's original spatial resolution. Lastly, the loop 697*77c1e3ccSAndroid Build Coastguard Workerrestoration is applied to resolve part of the high frequency lost. The last 698*77c1e3ccSAndroid Build Coastguard Workertwo steps together are called super-resolving process [5]. Similarly, decoding, 699*77c1e3ccSAndroid Build Coastguard Workerdeblocking and CDEF processes are applied at lower spatial resolution at 700*77c1e3ccSAndroid Build Coastguard Workerdecoder side. Then, the frames go through the super-resolving process. 701*77c1e3ccSAndroid Build Coastguard WorkerIn order to reduce overheads in line-buffers with respect to hardware 702*77c1e3ccSAndroid Build Coastguard Workerimplementation, the upscaling and downscaling process are applied to 703*77c1e3ccSAndroid Build Coastguard Workerhorizontal dimension only. 704*77c1e3ccSAndroid Build Coastguard Worker 705*77c1e3ccSAndroid Build Coastguard Worker### Film grain synthesis 706*77c1e3ccSAndroid Build Coastguard Worker 707*77c1e3ccSAndroid Build Coastguard WorkerAt encoder side, film grain is removed from the input video as a denoising 708*77c1e3ccSAndroid Build Coastguard Workerprocess. Then, the structure and intensity of the input video are analyzed 709*77c1e3ccSAndroid Build Coastguard Workerby canny edge detector, and smooth areas are used to estimate the strength 710*77c1e3ccSAndroid Build Coastguard Workerof film grain. Once the strength is estimated, the denoised video and film 711*77c1e3ccSAndroid Build Coastguard Workergrain parameters are sent to decoder side. Those parameters are used to 712*77c1e3ccSAndroid Build Coastguard Workersynthesis the grain and add it back to the decoded video, producing the final 713*77c1e3ccSAndroid Build Coastguard Workeroutput video. 714*77c1e3ccSAndroid Build Coastguard Worker 715*77c1e3ccSAndroid Build Coastguard WorkerIn order to reconstruct the film grain, the following parameters are sent to 716*77c1e3ccSAndroid Build Coastguard Workerdecoder side: lag value, autoregressive coefficients, values for precomputed 717*77c1e3ccSAndroid Build Coastguard Workerlook-up table index of chroma components, and a set of points for a piece-wise 718*77c1e3ccSAndroid Build Coastguard Workerlinear scaling function [6]. Those parameters are signaled as quantized 719*77c1e3ccSAndroid Build Coastguard Workerintegers including 64 bytes for scaling function and 74 bytes for 720*77c1e3ccSAndroid Build Coastguard Workerautoregressive coefficients. Once the parameters are received, an 721*77c1e3ccSAndroid Build Coastguard Workerautoregressive process is applied in a raster scan order to generate one 64x64 722*77c1e3ccSAndroid Build Coastguard Workerluma and two 32x32 chroma film grain templates [6]. Those templates are used 723*77c1e3ccSAndroid Build Coastguard Workerto generate the grain for the remaining part of a picture. 724*77c1e3ccSAndroid Build Coastguard Worker 725*77c1e3ccSAndroid Build Coastguard Worker## Screen content coding 726*77c1e3ccSAndroid Build Coastguard Worker 727*77c1e3ccSAndroid Build Coastguard WorkerTo improve the coding performance of screen content coding, the associated video 728*77c1e3ccSAndroid Build Coastguard Workercodec incorporates several coding tools,for example, intra block copy 729*77c1e3ccSAndroid Build Coastguard Worker(IntraBC) is employed to handle the repeated patterns in a screen picture, and 730*77c1e3ccSAndroid Build Coastguard Workerpalette mode is used to handle the screen blocks with a limited number of 731*77c1e3ccSAndroid Build Coastguard Workerdifferent colors. 732*77c1e3ccSAndroid Build Coastguard Worker 733*77c1e3ccSAndroid Build Coastguard Worker### Intra block copy 734*77c1e3ccSAndroid Build Coastguard Worker 735*77c1e3ccSAndroid Build Coastguard WorkerIntra Block Copy (IntraBC) [2] is a coding tool similar to inter-picture 736*77c1e3ccSAndroid Build Coastguard Workerprediction. The main difference is that in IntraBC, a predictor block is 737*77c1e3ccSAndroid Build Coastguard Workerformed from the reconstructed samples (before application of in-loop filtering) 738*77c1e3ccSAndroid Build Coastguard Workerof the current picture. Therefore, IntraBC can be considered as "motion 739*77c1e3ccSAndroid Build Coastguard Workercompensation" within current picture. 740*77c1e3ccSAndroid Build Coastguard Worker 741*77c1e3ccSAndroid Build Coastguard WorkerA block vector (BV) was coded to specify the location of the predictor block. 742*77c1e3ccSAndroid Build Coastguard WorkerThe BV precision is integer. The BV will be signalled in the bitstream since the 743*77c1e3ccSAndroid Build Coastguard Workerdecoder needs it to locate the predictor. For current block, the flag use 744*77c1e3ccSAndroid Build Coastguard WorkerIntraBC indicating whether current block is IntraBC mode is first transmitted in 745*77c1e3ccSAndroid Build Coastguard Workerbit stream. Then, if the current block is IntraBC mode, the BV difference diff 746*77c1e3ccSAndroid Build Coastguard Workeris obtained by subtracting the reference BV from the current BV, and then diff 747*77c1e3ccSAndroid Build Coastguard Workeris classified into four types according to the diff values of horizontal and 748*77c1e3ccSAndroid Build Coastguard Workervertical component. Type information needs to be transmitted into the bitstream, 749*77c1e3ccSAndroid Build Coastguard Workerafter that, diff values of two components may be signalled based on the type 750*77c1e3ccSAndroid Build Coastguard Workerinfo. 751*77c1e3ccSAndroid Build Coastguard Worker 752*77c1e3ccSAndroid Build Coastguard WorkerIntraBC is very effective for screen content coding, but it also brings a lot of 753*77c1e3ccSAndroid Build Coastguard Workerdifficulties to hardware design. To facilitate the hardware design, the 754*77c1e3ccSAndroid Build Coastguard Workerfollowing modifications are adopted. 755*77c1e3ccSAndroid Build Coastguard Worker 756*77c1e3ccSAndroid Build Coastguard Worker1) when IntraBC is allowed, the loop filters are disabled, which are de-blocking 757*77c1e3ccSAndroid Build Coastguard Workerfilter, the CDEF (Constrained Directional Enhancement Filter), and the Loop 758*77c1e3ccSAndroid Build Coastguard WorkerRestoration. By doing this, picture buffer of reconstructed samples can be 759*77c1e3ccSAndroid Build Coastguard Workershared between IntraBC and inter prediction. 760*77c1e3ccSAndroid Build Coastguard Worker 761*77c1e3ccSAndroid Build Coastguard Worker2) To facilitate parallel decoding, the prediction cannot exceed the restricted 762*77c1e3ccSAndroid Build Coastguard Workerareas. For one super block, if the coordinate of its top-left position is (x0, 763*77c1e3ccSAndroid Build Coastguard Workery0), the prediction at position (x, y) can be accessed by IntraBC, if y < y0 and 764*77c1e3ccSAndroid Build Coastguard Workerx < x0 + 2 * (y0 - y) 765*77c1e3ccSAndroid Build Coastguard Worker 766*77c1e3ccSAndroid Build Coastguard Worker3) To allow hardware writing back delay, immediate reconstructed areas cannot be 767*77c1e3ccSAndroid Build Coastguard Workeraccessed by IntraBC prediction. The restricted immediate reconstructed area can 768*77c1e3ccSAndroid Build Coastguard Workerbe 1 ∼ n super blocks. So on top of modification 2, if the coordinate of one 769*77c1e3ccSAndroid Build Coastguard Workersuper block's top-left position is (x0, y0), the prediction at position (x, y) 770*77c1e3ccSAndroid Build Coastguard Workercan be accessed by IntraBC, if y < y0 and x < x0 + 2 * (y0 - y) - D, where D 771*77c1e3ccSAndroid Build Coastguard Workerdenotes the restricted immediate reconstructed area. When D is one super block, 772*77c1e3ccSAndroid Build Coastguard Workerthe prediction area is shown in below figure. 773*77c1e3ccSAndroid Build Coastguard Worker 774*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\SCC_IntraBC.svg" alt="Intra block 775*77c1e3ccSAndroid Build Coastguard Workercopy" width="600" /> <figcaption>Figure 13: the prediction area for IntraBC mode 776*77c1e3ccSAndroid Build Coastguard Workerin one super block prediction</figcaption> </figure> 777*77c1e3ccSAndroid Build Coastguard Worker 778*77c1e3ccSAndroid Build Coastguard Worker### Palette mode 779*77c1e3ccSAndroid Build Coastguard Worker 780*77c1e3ccSAndroid Build Coastguard Worker# References 781*77c1e3ccSAndroid Build Coastguard Worker 782*77c1e3ccSAndroid Build Coastguard Worker[1] J. Han, Y. Xu and D. Mukherjee, "A butterfly structured design of the hybrid 783*77c1e3ccSAndroid Build Coastguard Workertransform coding scheme," 2013 Picture Coding Symposium (PCS), San Jose, CA, 784*77c1e3ccSAndroid Build Coastguard Worker2013, pp. 17-20.\ 785*77c1e3ccSAndroid Build Coastguard Worker[2] J. Li, H. Su, A. Converse, B. Li, R. Zhou, B. Lin, J. Xu, Y. Lu, and R. 786*77c1e3ccSAndroid Build Coastguard WorkerXiong, "Intra Block Copy for Screen Content in the Emerging AV1 Video Codec," 787*77c1e3ccSAndroid Build Coastguard Worker2018 Data Compression Conference, Snowbird, Utah, USA.\ 788*77c1e3ccSAndroid Build Coastguard Worker[3] S. Midtskogen and J.M. Valin. "The AV1 constrained directional enhancement 789*77c1e3ccSAndroid Build Coastguard Worker filter (CDEF)." In 2018 IEEE International Conference on Acoustics, Speech 790*77c1e3ccSAndroid Build Coastguard Worker and Signal Processing (ICASSP), pp. 1193-1197. IEEE, 2018.\ 791*77c1e3ccSAndroid Build Coastguard Worker[4] D. Mukherjee, S. Li, Y. Chen, A. Anis, S. Parker, and 792*77c1e3ccSAndroid Build Coastguard WorkerJ. Bankoski. "A switchable loop-restoration with side-information framework 793*77c1e3ccSAndroid Build Coastguard Workerfor the emerging AV1 video codec." In 2017 IEEE International Conference on 794*77c1e3ccSAndroid Build Coastguard WorkerImage Processing (ICIP), pp. 265-269. IEEE, 2017.\ 795*77c1e3ccSAndroid Build Coastguard Worker[5] Y. Chen, D. Murherjee, J. Han, A. Grange, Y. Xu, Z. Liu,... & C.H.Chiang, 796*77c1e3ccSAndroid Build Coastguard Worker(2018, June). "An overview of core coding tools in the AV1 video codec."" 797*77c1e3ccSAndroid Build Coastguard WorkerIn 2018 Picture Coding Symposium (PCS) (pp. 41-45). IEEE.\ 798*77c1e3ccSAndroid Build Coastguard Worker[6] A. Norkin, & N. Birkbeck, (2018, March). "Film grain synthesis for AV1 799*77c1e3ccSAndroid Build Coastguard Workervideo codec." In 2018 Data Compression Conference (pp. 3-12). IEEE. 800