Frame_num is not frame counter (it is not incremented after a non-reference frame) – benefits and rationale

Pls. bear in mind that the frame_num parameter in the slice header of AVC/H.264 picture is not actually the frame counter since frame_num is not incremented after a non-reference frame. So, if all frames in a video sequence are ued for reference then frame_num = frame counter.

For example in the following cadence (in encoding order)where subscript indexes denotes frame_num values and superscript indexes denotes POC, capital letter corresponds to frames used for reference:

     I₀⁰  P⁶₁    b²₂    b⁴₂   P⁸₂   P¹⁰₃                                                          (A)                                     

Here b²₂    b⁴₂  is a pair of B frames not used for reference.

• Benefit 1 [ Successive B-frames swapping]:

In the context of the above sequence (A)  let’s suppose that for keeping of HRD compliancy it’s worth to swap b²₂   and b⁴₂  (e.g. if  b²₂  is much greater than average B-frame size and CPB buffer violation estimated). Because frame_num values of the two successive B frames are equal and the two frames are independent each another, hence the swap does not violate decoding process but might prevent HRD or T-STD violations in some situations. So, an encoder can transmit the above cadence (A)  in a slightly different order:

     I₀⁰  P⁶₁    b⁴₂    b²₂   P⁸₂   P¹⁰₃

If frame_num had been incremented as frame counter we could not swap non-reference B-frames.

• Benefit 2 [ Stream thinning – removal of non-reference B-frames]:

Sometimes due to bandwidth problem a server can decide to thin transmitted stream by removal non-reference B-frames. Because a non-reference frame is not used for reference of next pictures, its removal does not impact on next pictures. Therefore removal of non-reference B-frames inevitable reduces the overall bitrate (although a jitter can be observed). Moreover, because frame_num is not incremented on non-reference frames, gaps in frame_num are avoided.  

If frame_num had been incremented as frame counter we could not remove non-reference frames without producing gaps in frame_num cadence.

• Benefit 3 [ Frame Rate Correction – duplication of frames]:

Sometimes  a frame rate correction is required from lower frame rate to higher one (e.g. from 30 fps to 60fps), in such case some frames should be duplicated.

Example: For the sake of simplicity let’s suppose that an IP-only (i.e. IPPPPP…) stream is given and it’s required to ‘upscale’ the frame rate from 29.97 fps to 30 fps. We can perform this task without re-encoding, by insertion of a non-reference P-skip frame each 300 frames. Notice that P-skip frame is a P-frame where all MBs are skipped (with zero motion vector).  Because P-skip frame is not used for reference its frame_num is not incremented and no gap in frame_num cadence created.