Reference List Reordering in H.264/AVC

Introduction

The purpose of the list reordering feature of H264/AVC is to show how reference frame reordering impacts on  coding efficiency.

This document describes in details how to make the reference list reordering in order to achieve better coding efficiency. The H.264/AVC supports special reference list reordering commands (signaled in the slice header) and which are applied on both reference lists: L0  (previous frames) and L1 (future B-frames) . The same reordering commands are performed in Encoder and Decoder, otherwise a drift (or async) between Encoder and Decoder is inevitable.

 Note: This shuffling by reordering commands is temporary and is valid only within a given frame (more precisely within a given slice).

The reference picture indexes  ref_idx_l0 and/or ref_idx_l1 (if B frames are used) is signaled for each MB inter partition, excepting the skip or the direct modes (in both skip and direct modes ref_idx is derived from previous values of ref_idx).

The parameter  ref_idx_l0[l1] is binarized as a truncated  Exp-Golomb code and then forwarded to Arithmetic Engine. Large values ref_idx_l0 or ref_idx_l1 require much more bins than low values. Indeed,  if the reference lists consists of  four references: Ref₀ (1 bin), Ref₁ (3 bins),  Ref₂ (3 bins) and Ref₃(5bins). 

When most the distant reference is more beneficial than the immediate previous one. The  situation of flash lights  is not uncommon, especially in gaming content. In cases of flash lights the reference Ref₃ is a good candidate for reference, despite its distance to the current frame, since the previous frames are “corrupted” by flash lights:

 

The problem is that Ref₃ requires 5 bins while Ref_o 1 bin. Hence Ref₃ index is bits-consuming. However, if we reorder the reference list for all slices within the current frame such and to place Ref₃ at the first position (with index 0 and 1 bin) and Ref₀ is placed in the last position respectively (the index 3 with 5 bins). In such case a gain in coding efficiency is expected. So, it’s worth to reorder the reference list as follows:

    Ref₃, Ref₁, Ref₂, Ref₀

Notice that reordering commands are signaled in the slice header, hence regions “corrupted” by flash lights are reordered.

 

 

Reference List Reordering Syntax

Reordering is signaled in the slice header with the following syntax elements (here the suffix ‘x’ denotes the refrence list L0 or L1)  :

ref_pic_list_modification_flag_lx –  if 1 then reordering commands are present in the slice header and are conducted by both Encoder and Decoder in the same way (to avoid a drift between Encoder and Decoder).

abs_diff_pic_num_minus1 

modification_of_pic_nums_idc  – together with abs_diff_pic_num_minus1  indicates which of the pictures in the reference list is moved. The value 3 means termination process.

 

 

Algorithm of Shuffling Reference Pictures

A tacit assumption, a frame_nums of all pictures in reference list are less than 2^{log2_max_frame_num}

 

// set initial prediction values if field_pic_flag==0    // i.e. the current picture is frame and not a field                        picNumPred = frame_num of current frame           Else  // fields, interlace video                    picNumPred = [frame_num of current frame]*2+1             RefIdxLX=0 Read modification_of_pic_nums_idc   // read first reordering command while (modification_of_pic_nums_idc!=3) // ‘3’ stands for termination            {                    Read abs_diff_pic_num_minus1         // compute frame_num of  frame to be moved          picNumToMove  =    picNumPred   – (abs_diff_pic_num_minus1 +1)                     Find the frame with frame_num = ‘picNumToMove’ in reference list /* place the frame picNumToMove at the slot with RefIdxLX, free the slot by  moving                            right-aligned reference pictures to right                           num_ref_idx_LX_active_minus1+1 is the size of reference list                       */                      RefPicListX  = 0  for( idx=num_ref_idx_LX_active_minus1+1; idx>RefIdxLX;idx–) RefPicListX[idx] = RefPicListX[idx-1]   RefPicListX[RefIdxLX] = reference frame with frame_num eq.  picNumToMove   Read modification_of_pic_nums_idc   // read next reordering command RefIdxLX = picNumToMove   // set the new pivot picNumPred = picNumToMove  // frame_num of moved picture            }

 

Example  (progressive case):

Let’s consider the following sequence in display and encoding orders (numbers denote POCs, POC incremented by 2 each frame, B frames not used for reference):

Display order:         I0⁰   b2¹    b4¹     P6¹    b8²    b10²   P12²

Encoding order:      I0⁰    P6¹    b2¹   b4¹    P12²   b8²     b10²

Note: superscripts are frame_num, frame _num is not incrmented if the previous frame is not used for reference

At P6¹ frame the reference list is L0={I0⁰}

At b2¹ and b4¹ frames  the reference lists: L0={I0⁰},  L1={ P6¹}

At P12² frame the reference list   L0={ P6¹,I0⁰}

 

Let’s suppose that for the frame P12² the reference I0⁰ is more beneficial, hence it’s worth to swap I0⁰ and P6¹ in the list L0 (because ref_idx=0 takes 1 bin and ref_idx=1 takes 3 bins). Therefore in each slice header of P12² the following parameters are signaled:

ref_pic_list_modification_flag_lx = 1       // reordering command is present

modification_of_pic_nums_idc = 0       // to address the frame P6¹ which should be moved and I0⁰ is inserted.

        modification_of_pic_nums_idc = 3    // signal to terminate  

Thus, the modified  reference list for  P12² is L0={ I0⁰, P6¹}, so the distant reference I0⁰ becomes the immediate.