Bitrate Adaptation: Stream Thinning and Dynamic Resolution Change

There are two main approaches to adapt bitrate:

1) Stream Thinning – removal non-reference frames

2) Bitrate Adaptation via Downscaling of input frames (or dynamic resolution methods)

 

Stream Thinning

Stream thinning (also called Bitrate Peeling or Frame Skipping) is a method to drop frames from the stream (in order to reduce bandwidth for example due to network congestion). The stream thinning method is simple and can be performed by Middle Boxes (e.g. a gateway), since neither deep parsing nor re-encoding required.

However, removal of encoded frames can incur so called drift errors and visual degradation. 

Example of drift errors due to removal of used for reference frame (drift errors are circled, the pictures are taken from the paper “Bit Rate Transcoding of H.264 Encoded Movies by Dropping Frames in the Compressed Domain” by Spyridon K. Kapotas et al., 2010)

 

Bitrate Adaptation via Downscaling (Dynamic Resolution)

if network bandwidth is lowered then one of methods is downscale of input frames, encode the downscaled frames, transmit with lower rate to a client. On the client’s side the stream is decoded and up-scaled (the upscaling is necessary, since the aspect ratio is commonly not preserved in downscaled images):

 

Note: In the paper  “Improving Streaming Quality and Bitrate Efficiency with Dynamic Resolution Selection” by Xavier Ducloux et al.  the method of Dynamic Resolution Selection is proposed and elaborated. Choosing correct per-scene resolution can save bandwidth without compromising visual quality:

 

For each scene different resolutions are checked and the minimal resolution passing a visually lossless barrier is chosen.  Basing on well-known empirical fact – around scene changes the visibility of visual distortions are reduced. Therefore the change of resolution at scene cuts apparently would not inflict visual quality flickering effect.

The above Dynamic Resolution method is not suited well for live video applications due to multiple iterations to find optimal resolution. 

However, the authors uses Machine Learning to find optimal resolution for each scene without iterations, this solution is tailored for live applications:

 

How to Drop Frames without Causing Visual Degradation?    

Answer:  Remove frames which are not used for reference.

Indeed, if unused for reference frame (usually B-frame) is discarded from the stream then no drift error is produced (because non-reference frame is not used for prediction of next pictures and its removal obviously does not impact on next pictures). It’s worth mentioning that video artifacts like stuttering might be observed (under some circumstances even if a single frame is dropped).

On the other hand, if used for reference frame is discarded from the stream, then drift errors are generated. The errors tend to accumulate and to cause video quality degradation until next IDR frame (or intra-refresh sequence) received to clean up the impairments. Even if drift errors are non-perceptible at the beginning they tend to be quickly accumulated and get very perceptible and annoying after a number of frames.

 

Use Case of Stream Thinning

Low-Delay with non-reference B-frames:

P1 and P2 are P-frames since they have only single reference (IDR).

B3,B4,B5 and B6 are general B-frames, they have two references and prediction can be bi-directional or uni-directional.

Removal of P1, B3 and B5 will not incur drift errors since these frames are not used for reference.

 

Stream thinning is closely related to temporal scaling, the following example shows how hierarchical GOP structure can be used for bandwidth adaptation:

IP frames constitute the basic layer – 15 fps

Addition of yellow B frames constitute 30 fps

Addition of green B frames constitute 60 fps

Thus, if you remove the green B-frames you reduce the frame rate (and the bandwidth respectively) from 60fps to 30fps.

 

However B-frames are not used in low-delay applications (due to reordering), the following example shows how to construct GOP with hierarchical P-frames: