xref: /aosp_15_r20/external/libaom/doc/AlgorithmDescription.md (revision 77c1e3ccc04c968bd2bc212e87364f250e820521)
1*77c1e3ccSAndroid Build Coastguard Worker<div style="font-size:3em; text-align:center;"> Algorithm Description </div>
2*77c1e3ccSAndroid Build Coastguard Worker
3*77c1e3ccSAndroid Build Coastguard Worker# Abstract
4*77c1e3ccSAndroid Build Coastguard WorkerThis document describes technical aspects of coding tools included in
5*77c1e3ccSAndroid Build Coastguard Workerthe associated codec. This document is not a specification of the associated
6*77c1e3ccSAndroid Build Coastguard Workercodec. Instead, it summarizes the highlighted features of coding tools for new
7*77c1e3ccSAndroid Build Coastguard Workerdevelopers. This document should be updated when significant new normative
8*77c1e3ccSAndroid Build Coastguard Workerchanges have been integrated into the associated codec.
9*77c1e3ccSAndroid Build Coastguard Worker
10*77c1e3ccSAndroid Build Coastguard Worker# Table of Contents
11*77c1e3ccSAndroid Build Coastguard Worker
12*77c1e3ccSAndroid Build Coastguard Worker[Abbreviations](#Abbreviations)
13*77c1e3ccSAndroid Build Coastguard Worker
14*77c1e3ccSAndroid Build Coastguard Worker[Algorithm description](#Algorithm-Description)
15*77c1e3ccSAndroid Build Coastguard Worker
16*77c1e3ccSAndroid Build Coastguard Worker- [Block Partitioning](#Block-Partitioning)
17*77c1e3ccSAndroid Build Coastguard Worker  - [Coding block partition](#Coding-block-partition)
18*77c1e3ccSAndroid Build Coastguard Worker  - [Transform block partition](#Transform-block-partition)
19*77c1e3ccSAndroid Build Coastguard Worker- [Intra Prediction](#Intra-Prediction)
20*77c1e3ccSAndroid Build Coastguard Worker  - [Directional intra prediction modes](#Directional-intra-prediction-modes)
21*77c1e3ccSAndroid Build Coastguard Worker  - [Non-directional intra prediction modes](#Non-directional-intra-prediction-modes)
22*77c1e3ccSAndroid Build Coastguard Worker  - [Recursive filtering modes](#Recursive-filtering-modes)
23*77c1e3ccSAndroid Build Coastguard Worker  - [Chroma from Luma mode](#Chroma-from-Luma-mode)
24*77c1e3ccSAndroid Build Coastguard Worker- [Inter Prediction](#Inter-Prediction)
25*77c1e3ccSAndroid Build Coastguard Worker  - [Motion vector prediction](#Motion-vector-prediction)
26*77c1e3ccSAndroid Build Coastguard Worker  - [Motion vector coding](#Motion-vector-coding)
27*77c1e3ccSAndroid Build Coastguard Worker  - [Interpolation filter for motion compensation](#Interpolation-filter-for-motion-compensation)
28*77c1e3ccSAndroid Build Coastguard Worker  - [Warped motion compensation](#Warped-motion-compensation)
29*77c1e3ccSAndroid Build Coastguard Worker  - [Overlapped block motion compensation](#Overlapped-block-motion-compensation)
30*77c1e3ccSAndroid Build Coastguard Worker  - [Reference frames](#Reference-frames)
31*77c1e3ccSAndroid Build Coastguard Worker  - [Compound Prediction](#Compound-Prediction)
32*77c1e3ccSAndroid Build Coastguard Worker- [Transform](#Transform)
33*77c1e3ccSAndroid Build Coastguard Worker- [Quantization](#Quantization)
34*77c1e3ccSAndroid Build Coastguard Worker- [Entropy Coding](#Entropy-Coding)
35*77c1e3ccSAndroid Build Coastguard Worker- [Loop filtering and post-processing](#Loop-filtering-and-post-processing)
36*77c1e3ccSAndroid Build Coastguard Worker  - [Deblocking](#Deblocking)
37*77c1e3ccSAndroid Build Coastguard Worker  - [Constrained directional enhancement](#Constrained-directional-enhancement)
38*77c1e3ccSAndroid Build Coastguard Worker  - [Loop Restoration filter](#Loop-Restoration-filter)
39*77c1e3ccSAndroid Build Coastguard Worker  - [Frame super-resolution](#Frame-super-resolution)
40*77c1e3ccSAndroid Build Coastguard Worker  - [Film grain synthesis](#Film-grain-synthesis)
41*77c1e3ccSAndroid Build Coastguard Worker- [Screen content coding](#Screen-content-coding)
42*77c1e3ccSAndroid Build Coastguard Worker  - [Intra block copy](#Intra-block-copy)
43*77c1e3ccSAndroid Build Coastguard Worker  - [Palette mode](#Palette-mode)
44*77c1e3ccSAndroid Build Coastguard Worker
45*77c1e3ccSAndroid Build Coastguard Worker[References](#References)
46*77c1e3ccSAndroid Build Coastguard Worker
47*77c1e3ccSAndroid Build Coastguard Worker# Abbreviations
48*77c1e3ccSAndroid Build Coastguard Worker
49*77c1e3ccSAndroid Build Coastguard WorkerCfL: Chroma from Luma\
50*77c1e3ccSAndroid Build Coastguard WorkerIntraBC: Intra block copy\
51*77c1e3ccSAndroid Build Coastguard WorkerLCU: Largest coding unit\
52*77c1e3ccSAndroid Build Coastguard WorkerOBMC: Overlapped Block Motion Compensation\
53*77c1e3ccSAndroid Build Coastguard WorkerCDEF: Constrained Directional Enhancement Filter
54*77c1e3ccSAndroid Build Coastguard Worker
55*77c1e3ccSAndroid Build Coastguard Worker# Algorithm Description
56*77c1e3ccSAndroid Build Coastguard Worker
57*77c1e3ccSAndroid Build Coastguard Worker## Block Partitioning
58*77c1e3ccSAndroid Build Coastguard Worker
59*77c1e3ccSAndroid Build Coastguard Worker### Coding block partition
60*77c1e3ccSAndroid Build Coastguard Worker
61*77c1e3ccSAndroid Build Coastguard WorkerThe largest coding block unit (LCU) applied in this codec is 128×128. In
62*77c1e3ccSAndroid Build Coastguard Workeraddition to no split mode `PARTITION_NONE`, the partition tree supports 9
63*77c1e3ccSAndroid Build Coastguard Workerdifferent partitioning patterns, as shown in below figure.
64*77c1e3ccSAndroid Build Coastguard Worker
65*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\partition_codingblock.svg"
66*77c1e3ccSAndroid Build Coastguard Workeralt="Partition" width="360" /> <figcaption>Figure 1: Supported coding block
67*77c1e3ccSAndroid Build Coastguard Workerpartitions</figcaption> </figure>
68*77c1e3ccSAndroid Build Coastguard Worker
69*77c1e3ccSAndroid Build Coastguard WorkerAccording to the number of sub-partitions, the 9 partition modes are summarized
70*77c1e3ccSAndroid Build Coastguard Workeras follows: 1. Four partitions: `PARTITION_SPLIT`, `PARTITION_VERT_4`,
71*77c1e3ccSAndroid Build Coastguard Worker`PARTITION_HORZ_4` 2. Three partitions (T-Shape): `PARTITION_HORZ_A`,
72*77c1e3ccSAndroid Build Coastguard Worker`PARTITION_HORZ_B`, `PARTITION_VERT_A`, `PARTITION_HORZ_B` 3. Two partitions:
73*77c1e3ccSAndroid Build Coastguard Worker`PARTITION_HORZ`, `PARTITION_VERT`
74*77c1e3ccSAndroid Build Coastguard Worker
75*77c1e3ccSAndroid Build Coastguard WorkerAmong all the 9 partitioning patterns, only `PARTITION_SPLIT` mode supports
76*77c1e3ccSAndroid Build Coastguard Workerrecursive partitioning, i.e., sub-partitions can be further split, other
77*77c1e3ccSAndroid Build Coastguard Workerpartitioning modes cannot further split. Particularly, for 8x8 and 128x128,
78*77c1e3ccSAndroid Build Coastguard Worker`PARTITION_VERT_4`, `PARTITION_HORZ_4` are not used, and for 8x8, T-Shape
79*77c1e3ccSAndroid Build Coastguard Workerpartitions are not used either.
80*77c1e3ccSAndroid Build Coastguard Worker
81*77c1e3ccSAndroid Build Coastguard Worker### Transform block partition
82*77c1e3ccSAndroid Build Coastguard Worker
83*77c1e3ccSAndroid Build Coastguard WorkerFor both intra and inter coded blocks, the coding block can be further
84*77c1e3ccSAndroid Build Coastguard Workerpartitioned into multiple transform units with the partitioning depth up to 2
85*77c1e3ccSAndroid Build Coastguard Workerlevels. The mapping from the transform size of the current depth to the
86*77c1e3ccSAndroid Build Coastguard Workertransform size of the next depth is shown in the following Table 1.
87*77c1e3ccSAndroid Build Coastguard Worker
88*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><figcaption>Table 1: Transform partition size
89*77c1e3ccSAndroid Build Coastguard Workersetting</figcaption> <img src="img\tx_partition.svg" alt="Partition" width="220"
90*77c1e3ccSAndroid Build Coastguard Worker/> </figure>
91*77c1e3ccSAndroid Build Coastguard Worker
92*77c1e3ccSAndroid Build Coastguard WorkerFurthermore, for intra coded blocks, the transform partition is done in a way
93*77c1e3ccSAndroid Build Coastguard Workerthat all the transform blocks have the same size, and the transform blocks are
94*77c1e3ccSAndroid Build Coastguard Workercoded in a raster scan order. An example of the transform block partitioning for
95*77c1e3ccSAndroid Build Coastguard Workerintra coded block is shown in the Figure 2.
96*77c1e3ccSAndroid Build Coastguard Worker
97*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\intra_tx_partition.svg"
98*77c1e3ccSAndroid Build Coastguard Workeralt="Partition" width="600" /> <figcaption>Figure 2: Example of transform
99*77c1e3ccSAndroid Build Coastguard Workerpartitioning for intra coded block</figcaption> </figure>
100*77c1e3ccSAndroid Build Coastguard Worker
101*77c1e3ccSAndroid Build Coastguard WorkerFor inter coded blocks, the transform unit partitioning can be done in a
102*77c1e3ccSAndroid Build Coastguard Workerrecursive manner with the partitioning depth up to 2 levels. The transform
103*77c1e3ccSAndroid Build Coastguard Workerpartitioning supports 1:1 (square), 1:2/2:1, and 1:4/4:1 transform unit sizes
104*77c1e3ccSAndroid Build Coastguard Workerranging from 4×4 to 64×64. If the coding block is smaller than or equal to
105*77c1e3ccSAndroid Build Coastguard Worker64x64, the transform block partitioning can only apply to luma component, for
106*77c1e3ccSAndroid Build Coastguard Workerchroma blocks, the transform block size is identical to the coding block size.
107*77c1e3ccSAndroid Build Coastguard WorkerOtherwise, if the coding block width or height is greater than 64, then both the
108*77c1e3ccSAndroid Build Coastguard Workerluma and chroma coding blocks will implicitly split into multiples of min(W,
109*77c1e3ccSAndroid Build Coastguard Worker64)x min(H, 64) and min(W, 32)x min(H, 32) transform blocks, respectively.
110*77c1e3ccSAndroid Build Coastguard Worker
111*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\inter_tx_partition.svg"
112*77c1e3ccSAndroid Build Coastguard Workeralt="Partition" width="400" /> <figcaption>Figure 3: Example of transform
113*77c1e3ccSAndroid Build Coastguard Workerpartitioning for inter coded block</figcaption> </figure>
114*77c1e3ccSAndroid Build Coastguard Worker
115*77c1e3ccSAndroid Build Coastguard Worker## Intra Prediction
116*77c1e3ccSAndroid Build Coastguard Worker
117*77c1e3ccSAndroid Build Coastguard Worker### Directional intra prediction modes
118*77c1e3ccSAndroid Build Coastguard Worker
119*77c1e3ccSAndroid Build Coastguard WorkerDirectional intra prediction modes are applied in intra prediction, which models
120*77c1e3ccSAndroid Build Coastguard Workerlocal textures using a given direction pattern. Directional intra prediction
121*77c1e3ccSAndroid Build Coastguard Workermodes are represented by nominal modes and angle delta. The nominal modes are
122*77c1e3ccSAndroid Build Coastguard Workersimilar set of intra prediction angles used in VP9, which includes 8 angles. The
123*77c1e3ccSAndroid Build Coastguard Workerindex value of angle delta is ranging from -3 ~ +3, and zero delta angle
124*77c1e3ccSAndroid Build Coastguard Workerindicates a nominal mode. The prediction angle is represented by a nominal intra
125*77c1e3ccSAndroid Build Coastguard Workerangle plus an angle delta. In total, there are 56 directional intra prediction
126*77c1e3ccSAndroid Build Coastguard Workermodes, as shown in the following figure. In the below figure, solid arrows
127*77c1e3ccSAndroid Build Coastguard Workerindicate directional intra prediction modes and dotted arrows represent non-zero
128*77c1e3ccSAndroid Build Coastguard Workerangle delta.
129*77c1e3ccSAndroid Build Coastguard Worker
130*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\intra_directional.svg"
131*77c1e3ccSAndroid Build Coastguard Workeralt="Directional intra" width="300" /> <figcaption>Figure 4: Directional intra
132*77c1e3ccSAndroid Build Coastguard Workerprediction modes</figcaption> </figure>
133*77c1e3ccSAndroid Build Coastguard Worker
134*77c1e3ccSAndroid Build Coastguard WorkerThe nominal mode index and angle delta index is signalled separately, and
135*77c1e3ccSAndroid Build Coastguard Workernominal mode index is signalled before the associated angle delta index. It is
136*77c1e3ccSAndroid Build Coastguard Workernoted that for small block sizes, where the coding gain from extending intra
137*77c1e3ccSAndroid Build Coastguard Workerprediction angles may saturate, only the nominal modes are used and angle delta
138*77c1e3ccSAndroid Build Coastguard Workerindex is not signalled.
139*77c1e3ccSAndroid Build Coastguard Worker
140*77c1e3ccSAndroid Build Coastguard Worker### Non-directional intra prediction modes
141*77c1e3ccSAndroid Build Coastguard Worker
142*77c1e3ccSAndroid Build Coastguard WorkerIn addition to directional intra prediction modes, four non-directional intra
143*77c1e3ccSAndroid Build Coastguard Workermodes which simulate smooth textures are also included. The four non-directional
144*77c1e3ccSAndroid Build Coastguard Workerintra modes include `SMOOTH_V`, `SMOOTH_H`, `SMOOTH` and `PAETH predictor`.
145*77c1e3ccSAndroid Build Coastguard Worker
146*77c1e3ccSAndroid Build Coastguard WorkerIn `SMOOTH V`, `SMOOTH H` and `SMOOTH modes`, the prediction values are
147*77c1e3ccSAndroid Build Coastguard Workergenerated using quadratic interpolation along vertical, horizontal directions,
148*77c1e3ccSAndroid Build Coastguard Workeror the average thereof. The samples used in the quadratic interpolation include
149*77c1e3ccSAndroid Build Coastguard Workerreconstructed samples from the top and left neighboring blocks and samples from
150*77c1e3ccSAndroid Build Coastguard Workerthe right and bottom boundaries which are approximated by top reconstructed
151*77c1e3ccSAndroid Build Coastguard Workersamples and the left reconstructed samples.
152*77c1e3ccSAndroid Build Coastguard Worker
153*77c1e3ccSAndroid Build Coastguard WorkerIn `PAETH predictor` mode, the prediction for each sample is assigned as one
154*77c1e3ccSAndroid Build Coastguard Workerfrom the top (T), left (L) and top-left (TL) reference samples, which has the
155*77c1e3ccSAndroid Build Coastguard Workervalue closest to the Paeth predictor value, i.e., T + L -TL. The samples used in
156*77c1e3ccSAndroid Build Coastguard Worker`PAETH predictor` are illustrated in below figure.
157*77c1e3ccSAndroid Build Coastguard Worker
158*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\intra_paeth.svg" alt="Directional
159*77c1e3ccSAndroid Build Coastguard Workerintra" width="300" /> <figcaption>Figure 5: Paeth predictor</figcaption>
160*77c1e3ccSAndroid Build Coastguard Worker</figure>
161*77c1e3ccSAndroid Build Coastguard Worker
162*77c1e3ccSAndroid Build Coastguard Worker### Recursive filtering modes
163*77c1e3ccSAndroid Build Coastguard Worker
164*77c1e3ccSAndroid Build Coastguard WorkerFive filtering intra modes are defined, and each mode specify a set of eight
165*77c1e3ccSAndroid Build Coastguard Worker7-tap filters. Given the selected filtering mode index (0~4), the current block
166*77c1e3ccSAndroid Build Coastguard Workeris divided into 4x2 sub-blocks. For one 4×2 sub-block, each sample is predicted
167*77c1e3ccSAndroid Build Coastguard Workerby 7-tap interpolation using the 7 top and left neighboring samples as inputs.
168*77c1e3ccSAndroid Build Coastguard WorkerDifferent filters are applied for samples located at different coordinates
169*77c1e3ccSAndroid Build Coastguard Workerwithin a 4×2 sub-block. The prediction process can be done recursively in unit
170*77c1e3ccSAndroid Build Coastguard Worker4x2 sub-block, which means that prediction samples generated for one 4x2
171*77c1e3ccSAndroid Build Coastguard Workerprediction block can be used to predict another 4x2 sub-block.
172*77c1e3ccSAndroid Build Coastguard Worker
173*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\intra_recursive.svg"
174*77c1e3ccSAndroid Build Coastguard Workeralt="Directional intra" width="300" /> <figcaption>Figure 6: Recursive filtering
175*77c1e3ccSAndroid Build Coastguard Workermodes</figcaption> </figure>
176*77c1e3ccSAndroid Build Coastguard Worker
177*77c1e3ccSAndroid Build Coastguard Worker### Chroma from Luma mode
178*77c1e3ccSAndroid Build Coastguard Worker
179*77c1e3ccSAndroid Build Coastguard WorkerChroma from Luma (CfL) is a chroma intra prediction mode, which models chroma
180*77c1e3ccSAndroid Build Coastguard Workersamples as a linear function of co-located reconstructed luma samples. To align
181*77c1e3ccSAndroid Build Coastguard Workerthe resolution between luma and chroma samples for different chroma sampling
182*77c1e3ccSAndroid Build Coastguard Workerformat, e.g., 4:2:0 and 4:2:2, reconstructed luma pixels may need to be
183*77c1e3ccSAndroid Build Coastguard Workersub-sampled before being used in CfL mode. In addition, the DC component is
184*77c1e3ccSAndroid Build Coastguard Workerremoved to form the AC contribution. In CfL mode, the model parameters which
185*77c1e3ccSAndroid Build Coastguard Workerspecify the linear function between two color components are optimized by
186*77c1e3ccSAndroid Build Coastguard Workerencoder signalled in the bitstream.
187*77c1e3ccSAndroid Build Coastguard Worker
188*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\intra_cfl.svg" alt="Directional
189*77c1e3ccSAndroid Build Coastguard Workerintra" width="700" /> <figcaption>Figure 7: CfL prediction</figcaption>
190*77c1e3ccSAndroid Build Coastguard Worker</figure>
191*77c1e3ccSAndroid Build Coastguard Worker
192*77c1e3ccSAndroid Build Coastguard Worker## Inter Prediction
193*77c1e3ccSAndroid Build Coastguard Worker
194*77c1e3ccSAndroid Build Coastguard Worker### Motion vector prediction
195*77c1e3ccSAndroid Build Coastguard Worker
196*77c1e3ccSAndroid Build Coastguard WorkerMotion vectors are predicted by neighboring blocks which can be either spatial
197*77c1e3ccSAndroid Build Coastguard Workerneighboring blocks, or temporal neighboring blocks located in a reference frame.
198*77c1e3ccSAndroid Build Coastguard WorkerA set of MV predictors will be identified by checking all these blocks and
199*77c1e3ccSAndroid Build Coastguard Workerutilized to encode the motion vector information.
200*77c1e3ccSAndroid Build Coastguard Worker
201*77c1e3ccSAndroid Build Coastguard Worker**Spatial motion vector prediction**
202*77c1e3ccSAndroid Build Coastguard Worker
203*77c1e3ccSAndroid Build Coastguard WorkerThere are two sets of spatial neighboring blocks that can be utilized for
204*77c1e3ccSAndroid Build Coastguard Workerfinding spatial MV predictors, including the adjacent spatial neighbors which
205*77c1e3ccSAndroid Build Coastguard Workerare direct top and left neighbors of the current block, and second outer spatial
206*77c1e3ccSAndroid Build Coastguard Workerneighbors which are close but not directly adjacent to the current block. The
207*77c1e3ccSAndroid Build Coastguard Workertwo sets of spatial neighboring blocks are illustrated in an example shown in
208*77c1e3ccSAndroid Build Coastguard WorkerFigure 8.
209*77c1e3ccSAndroid Build Coastguard Worker
210*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\inter_spatial_mvp.svg"
211*77c1e3ccSAndroid Build Coastguard Workeralt="Directional intra" width="350" /><figcaption>Figure 8: Motion field
212*77c1e3ccSAndroid Build Coastguard Workerestimation by linear projection</figcaption></figure>
213*77c1e3ccSAndroid Build Coastguard Worker
214*77c1e3ccSAndroid Build Coastguard WorkerFor each set of spatial neighbors, the top row will be checked from left to
215*77c1e3ccSAndroid Build Coastguard Workerright and then the left column will be checked from top to down. For the
216*77c1e3ccSAndroid Build Coastguard Workeradjacent spatial neighbors, an additional top-right block will be also checked
217*77c1e3ccSAndroid Build Coastguard Workerafter checking the left column neighboring blocks. For the non-adjacent spatial
218*77c1e3ccSAndroid Build Coastguard Workerneighbors, the top-left block located at (-1, -1) position will be checked
219*77c1e3ccSAndroid Build Coastguard Workerfirst, then the top row and left column in a similar manner as the adjacent
220*77c1e3ccSAndroid Build Coastguard Workerneighbors. The adjacent neighbors will be checked first, then the temporal MV
221*77c1e3ccSAndroid Build Coastguard Workerpredictor that will be described in the next subsection will be checked second,
222*77c1e3ccSAndroid Build Coastguard Workerafter that, the non-adjacent spatial neighboring blocks will be checked.
223*77c1e3ccSAndroid Build Coastguard Worker
224*77c1e3ccSAndroid Build Coastguard WorkerFor compound prediction which utilizes a pair of reference frames, the
225*77c1e3ccSAndroid Build Coastguard Workernon-adjacent spatial neighbors are not used for deriving the MV predictor.
226*77c1e3ccSAndroid Build Coastguard Worker
227*77c1e3ccSAndroid Build Coastguard Worker**Temporal motion vector prediction**
228*77c1e3ccSAndroid Build Coastguard Worker
229*77c1e3ccSAndroid Build Coastguard WorkerIn addition to spatial neighboring blocks, MV predictor can be also derived
230*77c1e3ccSAndroid Build Coastguard Workerusing co-located blocks of reference pictures, namely temporal MV predictor. To
231*77c1e3ccSAndroid Build Coastguard Workergenerate temporal MV predictor, the MVs of reference frames are first stored
232*77c1e3ccSAndroid Build Coastguard Workertogether with reference indices associated with the reference frame. Then for
233*77c1e3ccSAndroid Build Coastguard Workereach 8x8 block of the current frame, the MVs of a reference frame which pass the
234*77c1e3ccSAndroid Build Coastguard Worker8x8 block are identified and stored together with the reference frame index in a
235*77c1e3ccSAndroid Build Coastguard Workertemporal MV buffer. In an example shown in Figure 5, the MV of reference frame 1
236*77c1e3ccSAndroid Build Coastguard Worker(R1) pointing from R1 to a reference frame of R1 is identified, i.e., MVref,
237*77c1e3ccSAndroid Build Coastguard Workerwhich passes a 8x8 block (shaded in blue dots) of current frame. Then this MVref
238*77c1e3ccSAndroid Build Coastguard Workeris stored in the temporal MV buffer associated with this 8x8 block. <figure
239*77c1e3ccSAndroid Build Coastguard Workerclass="image"> <center><img src="img\inter_motion_field.svg" alt="Directional
240*77c1e3ccSAndroid Build Coastguard Workerintra" width="800" /><figcaption>Figure 9: Motion field estimation by linear
241*77c1e3ccSAndroid Build Coastguard Workerprojection</figcaption></figure> Finally, given a couple of pre-defined block
242*77c1e3ccSAndroid Build Coastguard Workercoordinates, the associated MVs stored in the temporal MV buffer are identified
243*77c1e3ccSAndroid Build Coastguard Workerand projected accordingly to derive a temporal MV predictor which points from
244*77c1e3ccSAndroid Build Coastguard Workerthe current block to its reference frame, e.g., MV0 in Figure 5. In Figure 6,
245*77c1e3ccSAndroid Build Coastguard Workerthe pre-defined block positions for deriving temporal MV predictors of a 16x16
246*77c1e3ccSAndroid Build Coastguard Workerblock are shown and up to 7 blocks will be checked to find valid temporal MV
247*77c1e3ccSAndroid Build Coastguard Workerpredictors.<figure class="image"> <center><img
248*77c1e3ccSAndroid Build Coastguard Workersrc="img\inter_tmvp_positions.svg" alt="Directional intra" width="300"
249*77c1e3ccSAndroid Build Coastguard Worker/><figcaption>Figure 10: Block positions for deriving temporal MV
250*77c1e3ccSAndroid Build Coastguard Workerpredictors</figcaption></figure> The temporal MV predictors are checked after
251*77c1e3ccSAndroid Build Coastguard Workerthe nearest spatial MV predictors but before the non-adjacent spatial MV
252*77c1e3ccSAndroid Build Coastguard Workerpredictors.
253*77c1e3ccSAndroid Build Coastguard Worker
254*77c1e3ccSAndroid Build Coastguard WorkerAll the spatial and temporal MV candidates will be put together in a pool, with
255*77c1e3ccSAndroid Build Coastguard Workereach predictor associated with a weighting determined during the scanning of the
256*77c1e3ccSAndroid Build Coastguard Workerspatial and temporal neighboring blocks. Based on the associated weightings, the
257*77c1e3ccSAndroid Build Coastguard Workercandidates are sorted and ranked, and up to four candidates will be used as a
258*77c1e3ccSAndroid Build Coastguard Workerlist MV predictor list.
259*77c1e3ccSAndroid Build Coastguard Worker
260*77c1e3ccSAndroid Build Coastguard Worker### Motion vector coding
261*77c1e3ccSAndroid Build Coastguard Worker
262*77c1e3ccSAndroid Build Coastguard Worker### Interpolation filter for motion compensation
263*77c1e3ccSAndroid Build Coastguard Worker
264*77c1e3ccSAndroid Build Coastguard Worker<mark>[Ed.: to be added]</mark>
265*77c1e3ccSAndroid Build Coastguard Worker
266*77c1e3ccSAndroid Build Coastguard Worker### Warped motion compensation
267*77c1e3ccSAndroid Build Coastguard Worker
268*77c1e3ccSAndroid Build Coastguard Worker**Global warped motion**
269*77c1e3ccSAndroid Build Coastguard Worker
270*77c1e3ccSAndroid Build Coastguard WorkerThe global motion information is signalled at each inter frame, wherein the
271*77c1e3ccSAndroid Build Coastguard Workerglobal motion type and motion parameters are included. The global motion types
272*77c1e3ccSAndroid Build Coastguard Workerand the number of the associated parameters are listed in the following table.
273*77c1e3ccSAndroid Build Coastguard Worker
274*77c1e3ccSAndroid Build Coastguard Worker
275*77c1e3ccSAndroid Build Coastguard Worker| Global motion type   | Number of parameters   |
276*77c1e3ccSAndroid Build Coastguard Worker|:------------------:|:--------------------:|
277*77c1e3ccSAndroid Build Coastguard Worker| Identity (zero motion)| 0 |
278*77c1e3ccSAndroid Build Coastguard Worker| Translation | 2 |
279*77c1e3ccSAndroid Build Coastguard Worker| Rotzoom  | 4 |
280*77c1e3ccSAndroid Build Coastguard Worker| General affine | 6 |
281*77c1e3ccSAndroid Build Coastguard Worker
282*77c1e3ccSAndroid Build Coastguard WorkerFor an inter coded block, after the reference frame index is
283*77c1e3ccSAndroid Build Coastguard Workertransmitted, if the motion of current block is indicated as global motion, the
284*77c1e3ccSAndroid Build Coastguard Workerglobal motion type and the associated parameters of the given reference will be
285*77c1e3ccSAndroid Build Coastguard Workerused for current block.
286*77c1e3ccSAndroid Build Coastguard Worker
287*77c1e3ccSAndroid Build Coastguard Worker**Local warped motion**
288*77c1e3ccSAndroid Build Coastguard Worker
289*77c1e3ccSAndroid Build Coastguard WorkerFor an inter coded block, local warped motion is allowed when the following
290*77c1e3ccSAndroid Build Coastguard Workerconditions are all satisfied:
291*77c1e3ccSAndroid Build Coastguard Worker
292*77c1e3ccSAndroid Build Coastguard Worker* Current block is single prediction
293*77c1e3ccSAndroid Build Coastguard Worker* Width or height is greater than or equal to 8 samples
294*77c1e3ccSAndroid Build Coastguard Worker* At least one of the immediate neighbors uses same reference frame with current block
295*77c1e3ccSAndroid Build Coastguard Worker
296*77c1e3ccSAndroid Build Coastguard WorkerIf the local warped motion is used for current block, instead of signalling the
297*77c1e3ccSAndroid Build Coastguard Workeraffine parameters, they are estimated by using mean square minimization of the
298*77c1e3ccSAndroid Build Coastguard Workerdistance between the reference projection and modeled projection based on the
299*77c1e3ccSAndroid Build Coastguard Workermotion vectors of current block and its immediate neighbors. To estimate the
300*77c1e3ccSAndroid Build Coastguard Workerparameters of local warped motion, the projection sample pair of the center
301*77c1e3ccSAndroid Build Coastguard Workerpixel in neighboring block and its corresponding pixel in the reference frame
302*77c1e3ccSAndroid Build Coastguard Workerare collected if the neighboring block uses the same reference frame with
303*77c1e3ccSAndroid Build Coastguard Workercurrent block. After that, 3 extra samples are created by shifting the center
304*77c1e3ccSAndroid Build Coastguard Workerposition by a quarter sample in one or two dimensions, and these samples are
305*77c1e3ccSAndroid Build Coastguard Workeralso considered as projection sample pairs to ensure the stability of the model
306*77c1e3ccSAndroid Build Coastguard Workerparameter estimation process.
307*77c1e3ccSAndroid Build Coastguard Worker
308*77c1e3ccSAndroid Build Coastguard Worker
309*77c1e3ccSAndroid Build Coastguard Worker### Overlapped block motion compensation
310*77c1e3ccSAndroid Build Coastguard Worker
311*77c1e3ccSAndroid Build Coastguard WorkerFor an inter-coded block, overlapped block motion compensation (OBMC) is allowed
312*77c1e3ccSAndroid Build Coastguard Workerwhen the following conditions are all satisfied.
313*77c1e3ccSAndroid Build Coastguard Worker
314*77c1e3ccSAndroid Build Coastguard Worker* Current block is single prediction
315*77c1e3ccSAndroid Build Coastguard Worker* Width or height is greater than or equal to 8 samples
316*77c1e3ccSAndroid Build Coastguard Worker* At least one of the neighboring blocks are inter-coded blocks
317*77c1e3ccSAndroid Build Coastguard Worker
318*77c1e3ccSAndroid Build Coastguard WorkerWhen OBMC is applied to current block, firstly, the initial inter prediction
319*77c1e3ccSAndroid Build Coastguard Workersamples is generated by using the assigned motion vector of current block, then
320*77c1e3ccSAndroid Build Coastguard Workerthe inter predicted samples for the current block and inter predicted samples
321*77c1e3ccSAndroid Build Coastguard Workerbased on motion vectors from the above and left blocks are blended to generate
322*77c1e3ccSAndroid Build Coastguard Workerthe final prediction samples.The maximum number of neighboring motion vectors is
323*77c1e3ccSAndroid Build Coastguard Workerlimited based on the size of current block, and up to 4 motion vectors from each
324*77c1e3ccSAndroid Build Coastguard Workerof upper and left blocks can be involved in the OBMC process of current block.
325*77c1e3ccSAndroid Build Coastguard Worker
326*77c1e3ccSAndroid Build Coastguard WorkerOne example of the processing order of neighboring blocks is shown in the
327*77c1e3ccSAndroid Build Coastguard Workerfollowing picture, wherein the values marked in each block indicate the
328*77c1e3ccSAndroid Build Coastguard Workerprocessing order of the motion vectors of current block and neighboring blocks.
329*77c1e3ccSAndroid Build Coastguard WorkerTo be specific, the motion vector of current block is firstly applied to
330*77c1e3ccSAndroid Build Coastguard Workergenerate inter prediction samples P0(x,y). Then motion vector of block 1 is
331*77c1e3ccSAndroid Build Coastguard Workerapplied to generate the prediction samples p1(x,y). After that, the prediction
332*77c1e3ccSAndroid Build Coastguard Workersamples in the overlapping area between block 0 and block 1 is an weighted
333*77c1e3ccSAndroid Build Coastguard Workeraverage of p0(x,y) and p1(x,y). The overlapping area of block 1 and block 0 is
334*77c1e3ccSAndroid Build Coastguard Workermarked in grey in the following picture. The motion vectors of block 2, 3, 4 are
335*77c1e3ccSAndroid Build Coastguard Workerfurther applied and blended in the same way.
336*77c1e3ccSAndroid Build Coastguard Worker
337*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\inter_obmc.svg" alt="Directional
338*77c1e3ccSAndroid Build Coastguard Workerintra" width="300" /><figcaption>Figure 11: neighboring blocks for OBMC
339*77c1e3ccSAndroid Build Coastguard Workerprocess</figcaption></figure>
340*77c1e3ccSAndroid Build Coastguard Worker
341*77c1e3ccSAndroid Build Coastguard Worker### Reference frames
342*77c1e3ccSAndroid Build Coastguard Worker
343*77c1e3ccSAndroid Build Coastguard Worker<mark>[Ed.: to be added]</mark>
344*77c1e3ccSAndroid Build Coastguard Worker
345*77c1e3ccSAndroid Build Coastguard Worker### Compound Prediction
346*77c1e3ccSAndroid Build Coastguard Worker
347*77c1e3ccSAndroid Build Coastguard Worker<mark>[Ed.: to be added]</mark>
348*77c1e3ccSAndroid Build Coastguard Worker
349*77c1e3ccSAndroid Build Coastguard Worker**Compound wedge prediction**
350*77c1e3ccSAndroid Build Coastguard Worker
351*77c1e3ccSAndroid Build Coastguard Worker<mark>[Ed.: to be added]</mark>
352*77c1e3ccSAndroid Build Coastguard Worker
353*77c1e3ccSAndroid Build Coastguard Worker**Difference-modulated masked prediction**
354*77c1e3ccSAndroid Build Coastguard Worker
355*77c1e3ccSAndroid Build Coastguard Worker<mark>[Ed.: to be added]</mark>
356*77c1e3ccSAndroid Build Coastguard Worker
357*77c1e3ccSAndroid Build Coastguard Worker**Frame distance-based compound prediction**
358*77c1e3ccSAndroid Build Coastguard Worker
359*77c1e3ccSAndroid Build Coastguard Worker<mark>[Ed.: to be added]</mark>
360*77c1e3ccSAndroid Build Coastguard Worker
361*77c1e3ccSAndroid Build Coastguard Worker**Compound inter-intra prediction**
362*77c1e3ccSAndroid Build Coastguard Worker
363*77c1e3ccSAndroid Build Coastguard Worker<mark>[Ed.: to be added]</mark>
364*77c1e3ccSAndroid Build Coastguard Worker
365*77c1e3ccSAndroid Build Coastguard Worker## Transform
366*77c1e3ccSAndroid Build Coastguard Worker
367*77c1e3ccSAndroid Build Coastguard WorkerThe separable 2D transform process is applied on prediction residuals. For the
368*77c1e3ccSAndroid Build Coastguard Workerforward transform, a 1-D vertical transform is performed first on each column of
369*77c1e3ccSAndroid Build Coastguard Workerthe input residual block, then a horizontal transform is performed on each row
370*77c1e3ccSAndroid Build Coastguard Workerof the vertical transform output. For the backward transform, a 1-D horizontal
371*77c1e3ccSAndroid Build Coastguard Workertransform is performed first on each row of the input de-quantized coefficient
372*77c1e3ccSAndroid Build Coastguard Workerblock, then a vertical transform is performed on each column of the horizontal
373*77c1e3ccSAndroid Build Coastguard Workertransform output. The primary 1-D transforms include four different types of
374*77c1e3ccSAndroid Build Coastguard Workertransform: a) 4-point, 8-point, 16-point, 32-point, 64-point DCT-2; b) 4-point,
375*77c1e3ccSAndroid Build Coastguard Worker8-point, 16-point asymmetric DST’s (DST-4, DST-7) and c) their flipped
376*77c1e3ccSAndroid Build Coastguard Workerversions; d) 4-point, 8-point, 16-point, 32-point identity transforms. When
377*77c1e3ccSAndroid Build Coastguard Workertransform size is 4-point, ADST refers to DST-7, otherwise, when transform size
378*77c1e3ccSAndroid Build Coastguard Workeris greater than 4-point, ADST refers to DST-4.
379*77c1e3ccSAndroid Build Coastguard Worker
380*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><figcaption>Table 2: Transform basis functions
381*77c1e3ccSAndroid Build Coastguard Worker(DCT-2, DST-4 and DST-7 for N-point input.</figcaption> <img src=
382*77c1e3ccSAndroid Build Coastguard Worker"img\tx_basis.svg" alt="Partition" width="450" /> </figure>
383*77c1e3ccSAndroid Build Coastguard Worker
384*77c1e3ccSAndroid Build Coastguard WorkerFor luma component, each transform block can select one pair of horizontal and
385*77c1e3ccSAndroid Build Coastguard Workervertical transform combination given a pre-defined set of transform type
386*77c1e3ccSAndroid Build Coastguard Workercandidates, and the selection is explicitly signalled into the bitstream.
387*77c1e3ccSAndroid Build Coastguard WorkerHowever, the selection is not signalled when Max(width,height) is 64. When
388*77c1e3ccSAndroid Build Coastguard Workerthe maximum of transform block width and height is greater than or equal to 32,
389*77c1e3ccSAndroid Build Coastguard Workerthe set of transform type candidates depend on the prediction mode, as described
390*77c1e3ccSAndroid Build Coastguard Workerin Table 3. Otherwise, when the maximum of transform block width and height is
391*77c1e3ccSAndroid Build Coastguard Workersmaller than 32, the set of transform type candidates depend on the prediction
392*77c1e3ccSAndroid Build Coastguard Workermode, as described in Table 4.
393*77c1e3ccSAndroid Build Coastguard Worker
394*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><figcaption>Table 3: Transform type candidates
395*77c1e3ccSAndroid Build Coastguard Workerfor luma component when max(width, height) is greater than or equal to 32.
396*77c1e3ccSAndroid Build Coastguard Worker</figcaption> <img src="img\tx_cands_large.svg" alt="Partition" width="370" />
397*77c1e3ccSAndroid Build Coastguard Worker</figure>
398*77c1e3ccSAndroid Build Coastguard Worker
399*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><figcaption>Table 4: Transform type candidates
400*77c1e3ccSAndroid Build Coastguard Workerfor luma component when max(width, height) is smaller than 32. </figcaption>
401*77c1e3ccSAndroid Build Coastguard Worker<img src="img\tx_cands_small.svg" alt="Partition" width="440" /> </figure>
402*77c1e3ccSAndroid Build Coastguard Worker
403*77c1e3ccSAndroid Build Coastguard WorkerThe set of transform type candidates (namely transform set) is defined in Table
404*77c1e3ccSAndroid Build Coastguard Worker5.
405*77c1e3ccSAndroid Build Coastguard Worker
406*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><figcaption>Table 5: Definition of transform set.
407*77c1e3ccSAndroid Build Coastguard Worker</figcaption> <img src="img\tx_set.svg" alt="Partition" width="450" /> </figure>
408*77c1e3ccSAndroid Build Coastguard Worker
409*77c1e3ccSAndroid Build Coastguard WorkerFor chroma component, the transform type selection is done in an implicit way.
410*77c1e3ccSAndroid Build Coastguard WorkerFor intra prediction residuals, the transform type is selected according to the
411*77c1e3ccSAndroid Build Coastguard Workerintra prediction mode, as specified in Table 4. For inter prediction residuals,
412*77c1e3ccSAndroid Build Coastguard Workerthe transform type is selected according to the transform type selection of the
413*77c1e3ccSAndroid Build Coastguard Workerco-located luma block. Therefore, for chroma component, there is no transform
414*77c1e3ccSAndroid Build Coastguard Workertype signalling in the bitstream.
415*77c1e3ccSAndroid Build Coastguard Worker
416*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><figcaption>Table 6: Transform type selection for
417*77c1e3ccSAndroid Build Coastguard Workerchroma component intra prediction residuals.</figcaption> <img src=
418*77c1e3ccSAndroid Build Coastguard Worker"img\tx_chroma.svg" alt="Partition" width="500" /> </figure>
419*77c1e3ccSAndroid Build Coastguard Worker
420*77c1e3ccSAndroid Build Coastguard WorkerThe computational cost of large size (e.g., 64-point) transforms is further
421*77c1e3ccSAndroid Build Coastguard Workerreduced by zeroing out all the coefficients except the following two cases:
422*77c1e3ccSAndroid Build Coastguard Worker
423*77c1e3ccSAndroid Build Coastguard Worker1. The top-left 32×32 quadrant for 64×64/64×32/32×64 DCT_DCT hybrid transforms
424*77c1e3ccSAndroid Build Coastguard Worker2. The left 32×16 area for 64×16 and top 16×32 for16×64 DCT_DCT hybrid transforms.
425*77c1e3ccSAndroid Build Coastguard Worker
426*77c1e3ccSAndroid Build Coastguard WorkerBoth the DCT-2 and ADST (DST-4, DST-7) are implemented using butterfly structure
427*77c1e3ccSAndroid Build Coastguard Worker[1], which included multiple stages of butterfly operations. Each butterfly
428*77c1e3ccSAndroid Build Coastguard Workeroperations can be calculated in parallel and different stages are cascaded in a
429*77c1e3ccSAndroid Build Coastguard Workersequential order.
430*77c1e3ccSAndroid Build Coastguard Worker
431*77c1e3ccSAndroid Build Coastguard Worker## Quantization
432*77c1e3ccSAndroid Build Coastguard WorkerQuantization of transform coefficients may apply different quantization step
433*77c1e3ccSAndroid Build Coastguard Workersize for DC and AC transform coefficients, and different quantization step size
434*77c1e3ccSAndroid Build Coastguard Workerfor luma and chroma transform coefficients. To specify the quantization step
435*77c1e3ccSAndroid Build Coastguard Workersize, in the frame header, a _**base_q_idx**_ syntax element is first signalled,
436*77c1e3ccSAndroid Build Coastguard Workerwhich is a 8-bit fixed length code specifying the quantization step size for
437*77c1e3ccSAndroid Build Coastguard Workerluma AC coefficients. The valid range of _**base_q_idx**_ is [0, 255].
438*77c1e3ccSAndroid Build Coastguard Worker
439*77c1e3ccSAndroid Build Coastguard WorkerAfter that, the delta value relative to base_q_idx for Luma DC coefficients,
440*77c1e3ccSAndroid Build Coastguard Workerindicated as DeltaQYDc is further signalled. Furthermore, if there are more than
441*77c1e3ccSAndroid Build Coastguard Workerone color plane, then a flag _**diff_uv_delta**_ is signaled to indicate whether
442*77c1e3ccSAndroid Build Coastguard WorkerCb and Cr color components apply different quantization index values. If
443*77c1e3ccSAndroid Build Coastguard Worker_**diff_uv_delta**_ is signalled as 0, then only the delta values relative to
444*77c1e3ccSAndroid Build Coastguard Workerbase_q_idx for chroma DC coefficients (indicated as DeltaQUDc) and AC
445*77c1e3ccSAndroid Build Coastguard Workercoefficients (indicated as DeltaQUAc) are signalled. Otherwise, the delta values
446*77c1e3ccSAndroid Build Coastguard Workerrelative to base_q_idx for both the Cb and Cr DC coefficients (indicated as
447*77c1e3ccSAndroid Build Coastguard WorkerDeltaQUDc and DeltaQVDc) and AC coefficients (indicated as DeltaQUAc and
448*77c1e3ccSAndroid Build Coastguard WorkerDeltaQVAc) are signalled.
449*77c1e3ccSAndroid Build Coastguard Worker
450*77c1e3ccSAndroid Build Coastguard WorkerThe above decoded DeltaQYDc, DeltaQUAc, DeltaQUDc, DeltaQVAc and DeltaQVDc are
451*77c1e3ccSAndroid Build Coastguard Workeradded to _base_q_idx_ to derive the quantization indices. Then these
452*77c1e3ccSAndroid Build Coastguard Workerquantization indices are further mapped to quantization step size according to
453*77c1e3ccSAndroid Build Coastguard Workertwo tables. For DC coefficients, the mapping from quantization index to
454*77c1e3ccSAndroid Build Coastguard Workerquantization step size for 8-bit, 10-bit and 12-bit internal bit depth is
455*77c1e3ccSAndroid Build Coastguard Workerspecified by a lookup table Dc_Qlookup[3][256], and the mapping from
456*77c1e3ccSAndroid Build Coastguard Workerquantization index to quantization step size for 8-bit, 10-bit and 12-bit is
457*77c1e3ccSAndroid Build Coastguard Workerspecified by a lookup table Ac_Qlookup[3][256].
458*77c1e3ccSAndroid Build Coastguard Worker
459*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\quant_dc.svg" alt="quant_dc"
460*77c1e3ccSAndroid Build Coastguard Workerwidth="800" /><figcaption>Figure 11: Quantization step size of DC coefficients
461*77c1e3ccSAndroid Build Coastguard Workerfor different internal bit-depth</figcaption></figure>
462*77c1e3ccSAndroid Build Coastguard Worker
463*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\quant_ac.svg" alt="quant_ac"
464*77c1e3ccSAndroid Build Coastguard Workerwidth="800" /><figcaption>Figure 12: Quantization step size of AC coefficients
465*77c1e3ccSAndroid Build Coastguard Workerfor different internal bit-depth</figcaption></figure>
466*77c1e3ccSAndroid Build Coastguard Worker
467*77c1e3ccSAndroid Build Coastguard WorkerGiven the quantization step size, indicated as _Q<sub>step_, the input quantized
468*77c1e3ccSAndroid Build Coastguard Workercoefficients is further de-quantized using the following formula:
469*77c1e3ccSAndroid Build Coastguard Worker
470*77c1e3ccSAndroid Build Coastguard Worker_F_ = sign * ( (_f_ * _Q<sub>step_) % 0xFFFFFF ) / _deNorm_
471*77c1e3ccSAndroid Build Coastguard Worker
472*77c1e3ccSAndroid Build Coastguard Worker, where _f_ is the input quantized coefficient, _F_ is the output dequantized
473*77c1e3ccSAndroid Build Coastguard Workercoefficient, _deNorm_ is a constant value derived from the transform block area
474*77c1e3ccSAndroid Build Coastguard Workersize, as indicated by the following table:
475*77c1e3ccSAndroid Build Coastguard Worker
476*77c1e3ccSAndroid Build Coastguard Worker| _deNorm_ | Tx block area size |
477*77c1e3ccSAndroid Build Coastguard Worker|----------|:--------------------------|
478*77c1e3ccSAndroid Build Coastguard Worker| 1| Less than 512 samples |
479*77c1e3ccSAndroid Build Coastguard Worker| 2 | 512 or 1024 samples |
480*77c1e3ccSAndroid Build Coastguard Worker| 4 | Greater than 1024 samples |
481*77c1e3ccSAndroid Build Coastguard Worker
482*77c1e3ccSAndroid Build Coastguard WorkerWhen the quantization index is 0, the quantization is performed using a
483*77c1e3ccSAndroid Build Coastguard Workerquantization step size equal to 1, which is lossless coding mode.
484*77c1e3ccSAndroid Build Coastguard Worker
485*77c1e3ccSAndroid Build Coastguard Worker## Entropy Coding
486*77c1e3ccSAndroid Build Coastguard Worker
487*77c1e3ccSAndroid Build Coastguard Worker**Entropy coding engine**
488*77c1e3ccSAndroid Build Coastguard Worker
489*77c1e3ccSAndroid Build Coastguard Worker<mark>[Ed.: to be added]</mark>
490*77c1e3ccSAndroid Build Coastguard Worker
491*77c1e3ccSAndroid Build Coastguard Worker**Coefficient coding**
492*77c1e3ccSAndroid Build Coastguard Worker
493*77c1e3ccSAndroid Build Coastguard WorkerFor each transform unit, the coefficient coding starts with coding a skip sign,
494*77c1e3ccSAndroid Build Coastguard Workerwhich is followed by the signaling of primary transform kernel type and the
495*77c1e3ccSAndroid Build Coastguard Workerend-of-block (EOB) position in case the transform coding is not skipped. After
496*77c1e3ccSAndroid Build Coastguard Workerthat, the coefficient values are coded in a multiple level map manner plus sign
497*77c1e3ccSAndroid Build Coastguard Workervalues. The level maps are coded as three level planes, namely lower-level,
498*77c1e3ccSAndroid Build Coastguard Workermiddle-level and higher-level planes, and the sign is coded as another separate
499*77c1e3ccSAndroid Build Coastguard Workerplane. The lower-level, middle-level and higher-level planes correspond to
500*77c1e3ccSAndroid Build Coastguard Workercorrespond to different ranges of coefficient magnitudes. The lower level plane
501*77c1e3ccSAndroid Build Coastguard Workercorresponds to the range of 0–2, the middle level plane takes care of the
502*77c1e3ccSAndroid Build Coastguard Workerrange of 3–14, and the higher-level plane covers the range of 15 and above.
503*77c1e3ccSAndroid Build Coastguard Worker
504*77c1e3ccSAndroid Build Coastguard WorkerThe three level planes are coded as follows. After the EOB position is coded,
505*77c1e3ccSAndroid Build Coastguard Workerthe lower-level and middle-level planes are coded together in backward scan
506*77c1e3ccSAndroid Build Coastguard Workerorder, and the scan order refers to zig-zag scan applied on the entire transform
507*77c1e3ccSAndroid Build Coastguard Workerunit basis. Then the sign plane and higher-level plane are coded together in
508*77c1e3ccSAndroid Build Coastguard Workerforward scan order. After that, the remainder (coefficient level minus 14) is
509*77c1e3ccSAndroid Build Coastguard Workerentropy coded using Exp-Golomb code.
510*77c1e3ccSAndroid Build Coastguard Worker
511*77c1e3ccSAndroid Build Coastguard WorkerThe context model applied to the lower level plane depends on the primary
512*77c1e3ccSAndroid Build Coastguard Workertransform directions, including: bi-directional, horizontal, and vertical, as
513*77c1e3ccSAndroid Build Coastguard Workerwell as transform size, and up to five neighbor (in frequency domain)
514*77c1e3ccSAndroid Build Coastguard Workercoefficients are used to derive the context. The middle level plane uses a
515*77c1e3ccSAndroid Build Coastguard Workersimilar context model, but the number of context neighbor coefficients is
516*77c1e3ccSAndroid Build Coastguard Workerreduced from 5 to 2. The higher-level plane is coded by Exp-Golomb code without
517*77c1e3ccSAndroid Build Coastguard Workerusing context model. For the sign plane, except the DC sign that is coded using
518*77c1e3ccSAndroid Build Coastguard Workerthe DC signs from its neighboring transform units, sign values of other
519*77c1e3ccSAndroid Build Coastguard Workercoefficients are coded directly without using context model.
520*77c1e3ccSAndroid Build Coastguard Worker
521*77c1e3ccSAndroid Build Coastguard Worker## Loop filtering and post-processing
522*77c1e3ccSAndroid Build Coastguard Worker
523*77c1e3ccSAndroid Build Coastguard Worker### Deblocking
524*77c1e3ccSAndroid Build Coastguard Worker
525*77c1e3ccSAndroid Build Coastguard WorkerThere are four methods when picking deblocking filter level, which are listed
526*77c1e3ccSAndroid Build Coastguard Workerbelow:
527*77c1e3ccSAndroid Build Coastguard Worker
528*77c1e3ccSAndroid Build Coastguard Worker* LPF_PICK_FROM_FULL_IMAGE: search the full image with different values
529*77c1e3ccSAndroid Build Coastguard Worker* LPF_PICK_FROM_Q: estimate the filter level based on quantizer and frame type
530*77c1e3ccSAndroid Build Coastguard Worker* LPF_PICK_FROM_SUBIMAGE: estimate the level from a portion of image
531*77c1e3ccSAndroid Build Coastguard Worker* LPF_PICK_MINIMAL_LPF: set the filter level to 0 and disable the deblocking
532*77c1e3ccSAndroid Build Coastguard Worker
533*77c1e3ccSAndroid Build Coastguard WorkerWhen estimating the filter level from the full image or sub-image, the searching
534*77c1e3ccSAndroid Build Coastguard Workerstarts from the previous frame filter level, ends when the filter step is less
535*77c1e3ccSAndroid Build Coastguard Workeror equal to zero. In addition to filter level, there are some other parameters
536*77c1e3ccSAndroid Build Coastguard Workerwhich control the deblocking filter such as sharpness level, mode deltas, and
537*77c1e3ccSAndroid Build Coastguard Workerreference deltas.
538*77c1e3ccSAndroid Build Coastguard Worker
539*77c1e3ccSAndroid Build Coastguard WorkerDeblocking is performed at 128x128 super block level, and the vertical and
540*77c1e3ccSAndroid Build Coastguard Workerhorizontal edges are filtered respectively. For a 128x128 super block, the
541*77c1e3ccSAndroid Build Coastguard Workervertical/horizontal edges aligned with each 8x8 block is firstly filtered. If
542*77c1e3ccSAndroid Build Coastguard Workerthe 4x4 transform is used, the internal edge aligned with a 4x4 block will be
543*77c1e3ccSAndroid Build Coastguard Workerfurther filtered. The filter length is switchable from 4-tap, 6-tap, 8-tap,
544*77c1e3ccSAndroid Build Coastguard Worker14-tap, and 0-tap (no filtering). The location of filter taps are identified
545*77c1e3ccSAndroid Build Coastguard Workerbased on the number of filter taps in order to compute the filter mask. When
546*77c1e3ccSAndroid Build Coastguard Workerfinally performing the filtering, outer taps are added if there is high edge
547*77c1e3ccSAndroid Build Coastguard Workervariance.
548*77c1e3ccSAndroid Build Coastguard Worker
549*77c1e3ccSAndroid Build Coastguard Worker### Constrained directional enhancement filter
550*77c1e3ccSAndroid Build Coastguard Worker
551*77c1e3ccSAndroid Build Coastguard Worker**Edge Direction Estimation**\
552*77c1e3ccSAndroid Build Coastguard WorkerIn CDEF, edge direction search is performed at 8x8 block-level. There are
553*77c1e3ccSAndroid Build Coastguard Workereight edge directions in total, as illustrated in Figure 13.
554*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\edge_direction.svg"
555*77c1e3ccSAndroid Build Coastguard Workeralt="Edge direction" width="700" /> <figcaption>Figure 13: Line number
556*77c1e3ccSAndroid Build Coastguard Workerk for pixels following direction d=0:7 in an 8x8 block.</figcaption> </figure>
557*77c1e3ccSAndroid Build Coastguard Worker
558*77c1e3ccSAndroid Build Coastguard WorkerThe optimal edge direction d_opt is found by maximizing the following
559*77c1e3ccSAndroid Build Coastguard Workerterm [3]:
560*77c1e3ccSAndroid Build Coastguard Worker
561*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\equ_edge_direction.svg"
562*77c1e3ccSAndroid Build Coastguard Workeralt="Equation edge direction" width="250" /> </figure>
563*77c1e3ccSAndroid Build Coastguard Worker<!-- $$d_{opt}=\max_{d} s_d$$
564*77c1e3ccSAndroid Build Coastguard Worker$$s_d = \sum_{k}\frac{1}{N_{d,k}}(\sum_{p\in P_{d,k}}x_p)^2,$$ -->
565*77c1e3ccSAndroid Build Coastguard Worker
566*77c1e3ccSAndroid Build Coastguard Workerwhere x_p is the value of pixel p, P_{d,k} is the set of pixels in
567*77c1e3ccSAndroid Build Coastguard Workerline k following direction d, N_{d,k} is the cardinality of P_{d,k}.
568*77c1e3ccSAndroid Build Coastguard Worker
569*77c1e3ccSAndroid Build Coastguard Worker**Directional filter**\
570*77c1e3ccSAndroid Build Coastguard WorkerCDEF consists two filter taps: the primary tap and the secondary tap.
571*77c1e3ccSAndroid Build Coastguard WorkerThe primary tap works along the edge direction (as shown in Figure 14),
572*77c1e3ccSAndroid Build Coastguard Workerwhile the secondary tap forms an oriented 45 degree off the edge direction
573*77c1e3ccSAndroid Build Coastguard Worker (as shown in Figure 15).
574*77c1e3ccSAndroid Build Coastguard Worker
575*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\primary_tap.svg"
576*77c1e3ccSAndroid Build Coastguard Workeralt="Primary tap" width="700" /> <figcaption>Figure 14: Primary filter
577*77c1e3ccSAndroid Build Coastguard Workertaps following edge direction. For even strengths a = 2 and b = 4, for
578*77c1e3ccSAndroid Build Coastguard Workerodd strengths a = 3 and b = 3. The filtered pixel is shown in the
579*77c1e3ccSAndroid Build Coastguard Workerhighlighted center.</figcaption> </figure>
580*77c1e3ccSAndroid Build Coastguard Worker
581*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\secondary_tap.svg"
582*77c1e3ccSAndroid Build Coastguard Workeralt="Edge direction" width="700" /> <figcaption>Figure 15: Secondary
583*77c1e3ccSAndroid Build Coastguard Workerfilter taps. The filtered pixel is shown in the highlighted center.
584*77c1e3ccSAndroid Build Coastguard Worker</figcaption> </figure>
585*77c1e3ccSAndroid Build Coastguard Worker
586*77c1e3ccSAndroid Build Coastguard WorkerCDEF can be described by the following equation:
587*77c1e3ccSAndroid Build Coastguard Worker
588*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\equ_dir_search.svg"
589*77c1e3ccSAndroid Build Coastguard Workeralt="Equation direction search" width="720" /> </figure>
590*77c1e3ccSAndroid Build Coastguard Worker
591*77c1e3ccSAndroid Build Coastguard Worker<!-- $$y(i,j)=x(i,j)+round(\sum_{m,n}w^{(p)}_{d,m,n}f(x(m,x)-x(i,j),S^{(p)},
592*77c1e3ccSAndroid Build Coastguard WorkerD)+\sum_{m,n}w^{(s)}_{d,m,n}f(x(m,x)-x(i,j),S^{(s)},D)),$$ -->
593*77c1e3ccSAndroid Build Coastguard Worker
594*77c1e3ccSAndroid Build Coastguard Workerwhere x(i,j) and y(i,j) are the input and output reconstructed values
595*77c1e3ccSAndroid Build Coastguard Workerof CDEF. p denotes primary tap, and s denotes secondary tap, w is
596*77c1e3ccSAndroid Build Coastguard Workerthe weight between primary and secondary tap. f(d,S,D) is a non-linear
597*77c1e3ccSAndroid Build Coastguard Workerfiltering function, S denotes filter strength, D is a damping parameter.
598*77c1e3ccSAndroid Build Coastguard WorkerFor 8-bit content, S^p ranges from 0 to 15, and S^s can be
599*77c1e3ccSAndroid Build Coastguard Worker0, 1, 2, or 4. D ranges from 3 to 6 for luma, and 2 to 4 for chroma.
600*77c1e3ccSAndroid Build Coastguard Worker
601*77c1e3ccSAndroid Build Coastguard Worker**Non linear filter**\
602*77c1e3ccSAndroid Build Coastguard WorkerCDEF uses a non-linear filtering function to prevent excessive blurring
603*77c1e3ccSAndroid Build Coastguard Workerwhen applied across an edge. It is achieved by ignoring pixels that are
604*77c1e3ccSAndroid Build Coastguard Workertoo different from the current pixels to be filtered. When the difference
605*77c1e3ccSAndroid Build Coastguard Workerbetween current pixel and it's neighboring pixel d is within a threshold,
606*77c1e3ccSAndroid Build Coastguard Workerf(d,S,D) = d, otherwise f(d,S,D) = 0. Specifically, the strength S
607*77c1e3ccSAndroid Build Coastguard Workerdetermines the maximum difference allowed and damping D determines the
608*77c1e3ccSAndroid Build Coastguard Workerpoint to ignore the filter tap.
609*77c1e3ccSAndroid Build Coastguard Worker
610*77c1e3ccSAndroid Build Coastguard Worker### Loop Restoration filter
611*77c1e3ccSAndroid Build Coastguard Worker
612*77c1e3ccSAndroid Build Coastguard Worker**Separable symmetric wiener filter**
613*77c1e3ccSAndroid Build Coastguard Worker
614*77c1e3ccSAndroid Build Coastguard WorkerLet F be a w x w 2D filter taps around the pixel to be filtered, denoted as
615*77c1e3ccSAndroid Build Coastguard Workera w^2 x 1 column vector. When compared with traditional Wiener Filter,
616*77c1e3ccSAndroid Build Coastguard WorkerSeparable Symmetric Wiener Filter has the following three constraints in order
617*77c1e3ccSAndroid Build Coastguard Workerto save signaling bits and reduce complexity [4]:
618*77c1e3ccSAndroid Build Coastguard Worker
619*77c1e3ccSAndroid Build Coastguard Worker1) The w x w filter window of is separated into horizontal and vertical w-tap
620*77c1e3ccSAndroid Build Coastguard Workerconvolutions.
621*77c1e3ccSAndroid Build Coastguard Worker
622*77c1e3ccSAndroid Build Coastguard Worker2) The horizontal and vertical filters are constrained to be symmetric.
623*77c1e3ccSAndroid Build Coastguard Worker
624*77c1e3ccSAndroid Build Coastguard Worker3) It is assumed that the summation of horizontal/vertical filter coefficients
625*77c1e3ccSAndroid Build Coastguard Workeris 1.
626*77c1e3ccSAndroid Build Coastguard Worker
627*77c1e3ccSAndroid Build Coastguard WorkerAs a result, F can be written as F = column_vectorize[ab^T], subject to a(i)
628*77c1e3ccSAndroid Build Coastguard Worker= a(w - 1 - i), b(i) = b(w - 1 - i), for i = [0, r - 1], and sum(a(i)) =
629*77c1e3ccSAndroid Build Coastguard Workersum(b(i)) = 1, where a is the vertical filters and b is the horizontal filters.
630*77c1e3ccSAndroid Build Coastguard WorkerThe derivation of the filters a and b starts from an initial guess of
631*77c1e3ccSAndroid Build Coastguard Workerhorizontal and vertical filters, optimizing one of the two while holding the
632*77c1e3ccSAndroid Build Coastguard Workerother fixed. In the implementation w = 7, thus, 3 taps need to be sent for
633*77c1e3ccSAndroid Build Coastguard Workerfilters a and b, respectively. When signaling the filter coefficients, 4, 5 and
634*77c1e3ccSAndroid Build Coastguard Worker6 bits are used for the first three filter taps, and the remaining ones are
635*77c1e3ccSAndroid Build Coastguard Workerobtained from the normalization and symmetry constraints. 30 bits in total are
636*77c1e3ccSAndroid Build Coastguard Workertransmitted for both vertical and horizontal filters.
637*77c1e3ccSAndroid Build Coastguard Worker
638*77c1e3ccSAndroid Build Coastguard Worker
639*77c1e3ccSAndroid Build Coastguard Worker**Dual self-guided filter**
640*77c1e3ccSAndroid Build Coastguard Worker
641*77c1e3ccSAndroid Build Coastguard WorkerDual self-guided filter is designed to firstly obtain two coarse restorations
642*77c1e3ccSAndroid Build Coastguard WorkerX1 and X2 of the degraded frame X, and the final restoration Xr is obtained as
643*77c1e3ccSAndroid Build Coastguard Workera combination of the degraded samples, and the difference between the degraded
644*77c1e3ccSAndroid Build Coastguard Workersamples and the coarse restorations [4]:
645*77c1e3ccSAndroid Build Coastguard Worker
646*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\equ_dual_self_guided.svg"
647*77c1e3ccSAndroid Build Coastguard Workeralt="Equation dual self guided filter" width="300" /> </figure>
648*77c1e3ccSAndroid Build Coastguard Worker<!-- $$X_r = X + \alpha (X_1 - X) + \beta (X_2 - X)$$ -->
649*77c1e3ccSAndroid Build Coastguard Worker
650*77c1e3ccSAndroid Build Coastguard WorkerAt encoder side, alpha and beta are computed using:
651*77c1e3ccSAndroid Build Coastguard Worker
652*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\equ_dual_self_para.svg"
653*77c1e3ccSAndroid Build Coastguard Workeralt="Equation dual self guided filter parameter" width="220" /> </figure>
654*77c1e3ccSAndroid Build Coastguard Worker<!-- $${\alpha, \beta}^T = (A^T A) ^{-1} A^T b,$$ -->
655*77c1e3ccSAndroid Build Coastguard Worker
656*77c1e3ccSAndroid Build Coastguard Workerwhere A = {X1 - X, X2 - X}, b = Y - X, and Y is the original source.
657*77c1e3ccSAndroid Build Coastguard Worker
658*77c1e3ccSAndroid Build Coastguard WorkerX1 and X2 are obtained using guided filtering, and the filtering is controlled
659*77c1e3ccSAndroid Build Coastguard Workerby a radius r and a noise parameter e, where a higher r implies a higher
660*77c1e3ccSAndroid Build Coastguard Workerspatial variance and a higher e implies a higher range variance [4]. X1 and X2
661*77c1e3ccSAndroid Build Coastguard Workercan be described by {r1, e1} and {r2, e2}, respectively.
662*77c1e3ccSAndroid Build Coastguard Worker
663*77c1e3ccSAndroid Build Coastguard WorkerThe encoder sends a 6-tuple {r1, e1, r2, e2, alpha, beta} to the decoder. In
664*77c1e3ccSAndroid Build Coastguard Workerthe implementation, {r1, e1, r2, e2} uses a 3-bit codebook, and {alpha, beta}
665*77c1e3ccSAndroid Build Coastguard Workeruses 7-bit each due to much higher precision, resulting in a total of 17 bits.
666*77c1e3ccSAndroid Build Coastguard Workerr is always less or equal to 3 [4].
667*77c1e3ccSAndroid Build Coastguard Worker
668*77c1e3ccSAndroid Build Coastguard WorkerGuided filtering can be described by a local linear model:
669*77c1e3ccSAndroid Build Coastguard Worker
670*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\equ_guided_filter.svg"
671*77c1e3ccSAndroid Build Coastguard Workeralt="Equation guided filter" width="155" /> </figure>
672*77c1e3ccSAndroid Build Coastguard Worker<!-- $$y=Fx+G,$$ -->
673*77c1e3ccSAndroid Build Coastguard Worker
674*77c1e3ccSAndroid Build Coastguard Workerwhere x and y are the input and output samples, F and G are determined by the
675*77c1e3ccSAndroid Build Coastguard Workerstatistics in the neighboring of the pixel to be filtered. It is called
676*77c1e3ccSAndroid Build Coastguard Workerself-guided filtering when the guidance image is the same as the degraded
677*77c1e3ccSAndroid Build Coastguard Workerimage[4].
678*77c1e3ccSAndroid Build Coastguard Worker
679*77c1e3ccSAndroid Build Coastguard WorkerFollowing are three steps when deriving F and G of the self-guided filtering:
680*77c1e3ccSAndroid Build Coastguard Worker
681*77c1e3ccSAndroid Build Coastguard Worker1) Compute mean u and variance d of pixels in a (2r + 1) x (2r + 1) window
682*77c1e3ccSAndroid Build Coastguard Workeraround the pixel to be filtered.
683*77c1e3ccSAndroid Build Coastguard Worker
684*77c1e3ccSAndroid Build Coastguard Worker2) For each pixel, compute f = d / (d + e); g = (1 - f)u.
685*77c1e3ccSAndroid Build Coastguard Worker
686*77c1e3ccSAndroid Build Coastguard Worker3) Compute F and G for each pixel as averages of f and g values in a 3 x 3
687*77c1e3ccSAndroid Build Coastguard Workerwindow around the pixel for use in step 2.
688*77c1e3ccSAndroid Build Coastguard Worker
689*77c1e3ccSAndroid Build Coastguard Worker### Frame super-resolution
690*77c1e3ccSAndroid Build Coastguard Worker
691*77c1e3ccSAndroid Build Coastguard WorkerIn order to improve the perceptual quality of decoded pictures, a
692*77c1e3ccSAndroid Build Coastguard Workersuper-resolution process is applied at low bit-rates [5]. First, at encoder
693*77c1e3ccSAndroid Build Coastguard Workerside, the source video is downscaled as a non-normative procedure. Second,
694*77c1e3ccSAndroid Build Coastguard Workerthe downscaled video is encoded, followed by deblocking and CDEF process.
695*77c1e3ccSAndroid Build Coastguard WorkerThird, a linear upscaling process is applied as a normative procedure to bring
696*77c1e3ccSAndroid Build Coastguard Workerthe encoded video back to it's original spatial resolution. Lastly, the loop
697*77c1e3ccSAndroid Build Coastguard Workerrestoration is applied to resolve part of the high frequency lost. The last
698*77c1e3ccSAndroid Build Coastguard Workertwo steps together are called super-resolving process [5]. Similarly, decoding,
699*77c1e3ccSAndroid Build Coastguard Workerdeblocking and CDEF processes are applied at lower spatial resolution at
700*77c1e3ccSAndroid Build Coastguard Workerdecoder side. Then, the frames go through the super-resolving process.
701*77c1e3ccSAndroid Build Coastguard WorkerIn order to reduce overheads in line-buffers with respect to hardware
702*77c1e3ccSAndroid Build Coastguard Workerimplementation, the upscaling and downscaling process are applied to
703*77c1e3ccSAndroid Build Coastguard Workerhorizontal dimension only.
704*77c1e3ccSAndroid Build Coastguard Worker
705*77c1e3ccSAndroid Build Coastguard Worker### Film grain synthesis
706*77c1e3ccSAndroid Build Coastguard Worker
707*77c1e3ccSAndroid Build Coastguard WorkerAt encoder side, film grain is removed from the input video as a denoising
708*77c1e3ccSAndroid Build Coastguard Workerprocess. Then, the structure and intensity of the input video are analyzed
709*77c1e3ccSAndroid Build Coastguard Workerby canny edge detector, and smooth areas are used to estimate the strength
710*77c1e3ccSAndroid Build Coastguard Workerof film grain. Once the strength is estimated, the denoised video and film
711*77c1e3ccSAndroid Build Coastguard Workergrain parameters are sent to decoder side. Those parameters are used to
712*77c1e3ccSAndroid Build Coastguard Workersynthesis the grain and add it back to the decoded video, producing the final
713*77c1e3ccSAndroid Build Coastguard Workeroutput video.
714*77c1e3ccSAndroid Build Coastguard Worker
715*77c1e3ccSAndroid Build Coastguard WorkerIn order to reconstruct the film grain, the following parameters are sent to
716*77c1e3ccSAndroid Build Coastguard Workerdecoder side: lag value, autoregressive coefficients, values for precomputed
717*77c1e3ccSAndroid Build Coastguard Workerlook-up table index of chroma components, and a set of points for a piece-wise
718*77c1e3ccSAndroid Build Coastguard Workerlinear scaling function [6]. Those parameters are signaled as quantized
719*77c1e3ccSAndroid Build Coastguard Workerintegers including 64 bytes for scaling function and 74 bytes for
720*77c1e3ccSAndroid Build Coastguard Workerautoregressive coefficients. Once the parameters are received, an
721*77c1e3ccSAndroid Build Coastguard Workerautoregressive process is applied in a raster scan order to generate one 64x64
722*77c1e3ccSAndroid Build Coastguard Workerluma and two 32x32 chroma film grain templates [6]. Those templates are used
723*77c1e3ccSAndroid Build Coastguard Workerto generate the grain for the remaining part of a picture.
724*77c1e3ccSAndroid Build Coastguard Worker
725*77c1e3ccSAndroid Build Coastguard Worker## Screen content coding
726*77c1e3ccSAndroid Build Coastguard Worker
727*77c1e3ccSAndroid Build Coastguard WorkerTo improve the coding performance of screen content coding, the associated video
728*77c1e3ccSAndroid Build Coastguard Workercodec incorporates several coding tools,for example, intra block copy
729*77c1e3ccSAndroid Build Coastguard Worker(IntraBC) is employed to handle the repeated patterns in a screen picture, and
730*77c1e3ccSAndroid Build Coastguard Workerpalette mode is used to handle the screen blocks with a limited number of
731*77c1e3ccSAndroid Build Coastguard Workerdifferent colors.
732*77c1e3ccSAndroid Build Coastguard Worker
733*77c1e3ccSAndroid Build Coastguard Worker### Intra block copy
734*77c1e3ccSAndroid Build Coastguard Worker
735*77c1e3ccSAndroid Build Coastguard WorkerIntra Block Copy (IntraBC) [2] is a coding tool similar to inter-picture
736*77c1e3ccSAndroid Build Coastguard Workerprediction. The main difference is that in IntraBC, a predictor block is
737*77c1e3ccSAndroid Build Coastguard Workerformed from the reconstructed samples (before application of in-loop filtering)
738*77c1e3ccSAndroid Build Coastguard Workerof the current picture. Therefore, IntraBC can be considered as "motion
739*77c1e3ccSAndroid Build Coastguard Workercompensation" within current picture.
740*77c1e3ccSAndroid Build Coastguard Worker
741*77c1e3ccSAndroid Build Coastguard WorkerA block vector (BV) was coded to specify the location of the predictor block.
742*77c1e3ccSAndroid Build Coastguard WorkerThe BV precision is integer. The BV will be signalled in the bitstream since the
743*77c1e3ccSAndroid Build Coastguard Workerdecoder needs it to locate the predictor. For current block, the flag use
744*77c1e3ccSAndroid Build Coastguard WorkerIntraBC indicating whether current block is IntraBC mode is first transmitted in
745*77c1e3ccSAndroid Build Coastguard Workerbit stream. Then, if the current block is IntraBC mode, the BV difference diff
746*77c1e3ccSAndroid Build Coastguard Workeris obtained by subtracting the reference BV from the current BV, and then diff
747*77c1e3ccSAndroid Build Coastguard Workeris classified into four types according to the diff values of horizontal and
748*77c1e3ccSAndroid Build Coastguard Workervertical component. Type information needs to be transmitted into the bitstream,
749*77c1e3ccSAndroid Build Coastguard Workerafter that, diff values of two components may be signalled based on the type
750*77c1e3ccSAndroid Build Coastguard Workerinfo.
751*77c1e3ccSAndroid Build Coastguard Worker
752*77c1e3ccSAndroid Build Coastguard WorkerIntraBC is very effective for screen content coding, but it also brings a lot of
753*77c1e3ccSAndroid Build Coastguard Workerdifficulties to hardware design. To facilitate the hardware design, the
754*77c1e3ccSAndroid Build Coastguard Workerfollowing modifications are adopted.
755*77c1e3ccSAndroid Build Coastguard Worker
756*77c1e3ccSAndroid Build Coastguard Worker1) when IntraBC is allowed, the loop filters are disabled, which are de-blocking
757*77c1e3ccSAndroid Build Coastguard Workerfilter, the CDEF (Constrained Directional Enhancement Filter), and the Loop
758*77c1e3ccSAndroid Build Coastguard WorkerRestoration. By doing this, picture buffer of reconstructed samples can be
759*77c1e3ccSAndroid Build Coastguard Workershared between IntraBC and inter prediction.
760*77c1e3ccSAndroid Build Coastguard Worker
761*77c1e3ccSAndroid Build Coastguard Worker2) To facilitate parallel decoding, the prediction cannot exceed the restricted
762*77c1e3ccSAndroid Build Coastguard Workerareas. For one super block, if the coordinate of its top-left position is (x0,
763*77c1e3ccSAndroid Build Coastguard Workery0), the prediction at position (x, y) can be accessed by IntraBC, if y < y0 and
764*77c1e3ccSAndroid Build Coastguard Workerx < x0 + 2 * (y0 - y)
765*77c1e3ccSAndroid Build Coastguard Worker
766*77c1e3ccSAndroid Build Coastguard Worker3) To allow hardware writing back delay, immediate reconstructed areas cannot be
767*77c1e3ccSAndroid Build Coastguard Workeraccessed by IntraBC prediction. The restricted immediate reconstructed area can
768*77c1e3ccSAndroid Build Coastguard Workerbe 1 ∼ n super blocks. So on top of modification 2, if the coordinate of one
769*77c1e3ccSAndroid Build Coastguard Workersuper block's top-left position is (x0, y0), the prediction at position (x, y)
770*77c1e3ccSAndroid Build Coastguard Workercan be accessed by IntraBC, if y < y0 and x < x0 + 2 * (y0 - y) - D, where D
771*77c1e3ccSAndroid Build Coastguard Workerdenotes the restricted immediate reconstructed area. When D is one super block,
772*77c1e3ccSAndroid Build Coastguard Workerthe prediction area is shown in below figure.
773*77c1e3ccSAndroid Build Coastguard Worker
774*77c1e3ccSAndroid Build Coastguard Worker<figure class="image"> <center><img src="img\SCC_IntraBC.svg" alt="Intra block
775*77c1e3ccSAndroid Build Coastguard Workercopy" width="600" /> <figcaption>Figure 13: the prediction area for IntraBC mode
776*77c1e3ccSAndroid Build Coastguard Workerin one super block prediction</figcaption> </figure>
777*77c1e3ccSAndroid Build Coastguard Worker
778*77c1e3ccSAndroid Build Coastguard Worker### Palette mode
779*77c1e3ccSAndroid Build Coastguard Worker
780*77c1e3ccSAndroid Build Coastguard Worker# References
781*77c1e3ccSAndroid Build Coastguard Worker
782*77c1e3ccSAndroid Build Coastguard Worker[1] J. Han, Y. Xu and D. Mukherjee, "A butterfly structured design of the hybrid
783*77c1e3ccSAndroid Build Coastguard Workertransform coding scheme," 2013 Picture Coding Symposium (PCS), San Jose, CA,
784*77c1e3ccSAndroid Build Coastguard Worker2013, pp. 17-20.\
785*77c1e3ccSAndroid Build Coastguard Worker[2] J. Li, H. Su, A. Converse, B. Li, R. Zhou, B. Lin, J. Xu, Y. Lu, and R.
786*77c1e3ccSAndroid Build Coastguard WorkerXiong, "Intra Block Copy for Screen Content in the Emerging AV1 Video Codec,"
787*77c1e3ccSAndroid Build Coastguard Worker2018 Data Compression Conference, Snowbird, Utah, USA.\
788*77c1e3ccSAndroid Build Coastguard Worker[3] S. Midtskogen and J.M. Valin. "The AV1 constrained directional enhancement
789*77c1e3ccSAndroid Build Coastguard Worker filter (CDEF)." In 2018 IEEE International Conference on Acoustics, Speech
790*77c1e3ccSAndroid Build Coastguard Worker  and Signal Processing (ICASSP), pp. 1193-1197. IEEE, 2018.\
791*77c1e3ccSAndroid Build Coastguard Worker[4] D. Mukherjee, S. Li, Y. Chen, A. Anis, S. Parker, and
792*77c1e3ccSAndroid Build Coastguard WorkerJ. Bankoski. "A switchable loop-restoration with side-information framework
793*77c1e3ccSAndroid Build Coastguard Workerfor the emerging AV1 video codec." In 2017 IEEE International Conference on
794*77c1e3ccSAndroid Build Coastguard WorkerImage Processing (ICIP), pp. 265-269. IEEE, 2017.\
795*77c1e3ccSAndroid Build Coastguard Worker[5] Y. Chen, D. Murherjee, J. Han, A. Grange, Y. Xu, Z. Liu,... & C.H.Chiang,
796*77c1e3ccSAndroid Build Coastguard Worker(2018, June). "An overview of core coding tools in the AV1 video codec.""
797*77c1e3ccSAndroid Build Coastguard WorkerIn 2018 Picture Coding Symposium (PCS) (pp. 41-45). IEEE.\
798*77c1e3ccSAndroid Build Coastguard Worker[6] A. Norkin, & N. Birkbeck, (2018, March). "Film grain synthesis for AV1
799*77c1e3ccSAndroid Build Coastguard Workervideo codec." In 2018 Data Compression Conference (pp. 3-12). IEEE.
800