Classification Visual Artifacts in Video Coding Systems

Content

Blurring

Color Bleeding

Flickering

Mosquito Noise

      Ringing

Blockiness

Staircase

          Motion Blur

      Tearing

 

General

Visual artifacts such as ringing, mosquito noise, flickering, blurring, blockiness etc. are caused directly or indirectly by the quantization in an encoder.

Two types of video distortions are present: compression artifacts (due to lossy compression) and scaling artifacts (for lower bitrates, video is downsampled before compression, and later upsampled on the viewer’s device).

The quantization is the main source of information leakage (the information can be also leaked via in-loop deblocking filter when a true edge is mistakenly eliminated).  It’s worth mentioning that there are other visual distortions (not related to video coding) and caused by digital cameras (e.g. motion blur).

There exist a wide family of filters to suppress video coding artefacts: de-ringing, de-mosquito etc. Some of these filters are applied after decoding, some are a part of modern video coding standards (so called in-loop filters). For example, the deblocking filter is a part of the reconstructed path of almost all video coding standards.

In this short note I describe a limited number of visual artifacts, reasons for their appearance.

There are other visual impairments, e.g. stuttering which  related to networking (non-uniform arrivals of images).

 

Blurring 

Blurring is a reduction of the sharpness of an image by blending the colors of neighboring pixels. Blurring is a result of coarse quantization (low bitrate scenarios) when most of high-frequency components in the transform domain are quantized to zero. As a result, edges get speared and high-detailed patterns are eliminated (e.g. a grass appears as a green plasticine). An image appears blurred when its high spatial frequency values in the spectrum are attenuated by the quantization. Notice that there the blur can be inserted by camera – the motion blur (see below).

Generally speaking, the coarse quantization works as a strong low-pass filter. On a display’s side a de-blur filter can be applied to sharpen existing edges, however high-detailed patterns are lost forever.

One of methods to estimate the sharpness of an image consists of two steps:

• Apply Laplacian filter to enhance edges
• Compute the variance of the filtered image

The script sharpness.py   computes the sharpness according to the above algorithm.

Usage:

-i                               input yuv-file (yuv420p)
–width                      frame width (must be even)
–height                     frame height (must be even)
-k                              frame index in the yuv-file to compute the sharpness, default 0
-v                              verbose mode show original and edge images (default false)

Example [ compute the sharpness for 10-th frame]

python sharpness.py -i  battlefield_384x320.raw --width 384 --height 320 -k 10
sharpness 155.25

Sharpness 155.25:

 

Sharpness 962:

 

 

Color Bleeding 

The visual artifact awkwardly named as ‘color bleeding’ is not a result of quantization. It’s caused by chroma-subsampling of RGB images (computer generated images) to 4:2:0 format.

Owing to sub-sampling chroma samples of a foreground object cross  the object’s boundaries. The appearance of color bleeding tends to be prominently visible on color-saturated objects:

 

Flickering 

Flickering is a periodic visual distortion – a noticeable discontinuity in perceptual quality between an intra frame (I-frame) and its preceding/following inter frames (known as the flicker or I-pulsing).

In most scenarios I-frames are periodically inserted to support random access. I-frames tend to be much greater than P/B frames to compensate absence of  temporal redundancy exploitation.It’s not uncommon that encoders undershoot or overshoot in visual quality of I-frames and consequently I-frames got worse or better in visual quality relative neighboring pictures.  Therefore such abrupt fluctuations in visual quality can be visible and annoying.

Intra-refresh methods is one of ways to cope with the flickering.

There is another reason for the flicker (mainly aperiodic) – abundant use of skip modes in inter frames (especially in low-bitrate and in fast presets).

Let’s formulate mathematically what’s the flicker:

• O_t an image at time t in the original video
• P_t the predicted image used by the decoder, either motion compensated or intra predicted

At the encoder’s side residual (O_t  – P_t) is transformed (the transform matrix is denoted by T) and quantized (denoted by Q) before being entropy coded:

Q• T• (O_t  – P_t)

The decoder reverses the above process to get the reconstructed image R_t :

R_t   = T^-1 •Q^-1 • Q• T• (O_t  – P_t) + P_t

From the assumption that  O_t  ≈ O_t-1   (a still scene) does not follow with the necessity that R_t  ≈ R_t-1  , since P_t  and P_t-1 can significantly differ (e.g. a significant portion of MBs chosen as skipped). This can result in a discontinuity, which is observed as the flicker.  In case  abundance of skip MBs,  R_t   = P_t  for skipped block and small differences are ignored and accumulated.

One of techniques to attenuate the I-frame based flicker in playback is to build the motion compensated image X_I for the I-frame (e.g. by using block-matching) and to average X_I with R_t. However, this method might generate much severe visual impairments like ‘ghosts’ since the block-matching does not always find true motion. Moreover, in case of occlusion some areas in a picture can be absent of reference.

More reliable method to reduce flickering is described in  the paper “POST-PROCESSING FOR FLICKER REDUCTION IN H.264/AVC”,  Yair Kuszpet et al.

 

i provide outline of the method:

Assumption: it was observed that flicker is more common in smooth areas of a frame. Therefore ‘de-flicker filter’ should be applied on smooth areas. 

 

For each received I-frame R_I Do:

• • Generate Motion Compensated Image –  X_I

• • Identifying smooth or flat areas (by generation of smoothness pixel-based map)

• • Apply Morphological Opening on the smoothness map to enhance smoothed clusters

• • Generate Flat-Difference map by multiplying the smoothness score of each pixel with its the absolute difference between R_I and X_I

• • Generate the flicker map (the flicker map shows the areas of the intra frame that exhibit flicker)

• • Apply De-flickering on ‘smooth’ pixel (i,j) by  X_ij+(1-α)R_i,j , where α is the value in the range [0..1].

 

Note: According to the paper “On the visibility of flicker distortions in naturalistic videos“, by L. K. Choi, L. K. Cormack, and A. C. Bovik,  Jul. 2013, large object motions strongly suppress the visibility of local flicker.

Mosquito Noise

Mosquito noise gets its name from resembling a mosquito flying around a person’s head and shoulders.

 

To suppress mosquito noise on the encoder side it’s recommended to use small prediction blocks (e.g. 4×4). Although, active using of 4×4 blocks can deteriorate the coding efficiency.

In HEVC, in-loop filter SAO effectively reduces mosquito noise:

 

Ringing 

Ringing artifacts manifest themselves in the form of a “halo,” band, or “ghost” near sharp edges:

 

The ringing artifact is associated with the Gibbs phenomenon (it is a direct result of quantization leading to high-frequency irregularities in the reconstruction). The ringing is more evident along high contrast edges within smooth areas.

On the encoder’s side to suppress ringing it’s recommended to use small transform blocks across contrast edges “immersed” in smooth areas.

 

Blockiness

Blockiness artifact is characterized by an artificial block pattern visible in the picture. It is due to the independent quantization of individual blocks in block-based DCT coding schemes, leading to discontinuities at the boundaries of adjacent blocks. The blocking effect is often the most visible and annoying artifact in a compressed video, given its periodicity and the extent of the pattern. More modern codecs, like the H.264/AVC, use an in-loop deblocking filter to suppress blockiness.

 

Staircase and basis pattern artifacts

Another visual artifact closely related to ringing is the so-called staircase artifact which refers to the incapability of horizontal and vertical basis DCT or DST functions to accurately represent diagonal edges (similar to steep edges)

The appearance of a diagonal edge may be similar to the pattern of a staircase rather than that of a smooth diagonal connection between two points.

High quantization tends to reduce the number of non-zero coefficients in a transformed block, boosting visual appearance of  basis pattern artifacts.

 

Motion Blur

Motion blur  appears in scenes of fast moving objects when during camera’s exposure an object makes a significant movement. This phenomenon is absent in computer generated graphics:

the picture is taken from the book “Digital Video, Concepts, Methods, and Metrics” by Shahriar Akramullah

Motion blur can be eliminated with increase of the frame rate – a moving ball may be blurry at 30 fps, but would be visible clear at 120 fps (although most of TV sets don’t support 12o fps, only 60fps).

Another way to avoid the motion blur is by panning the camera to track the moving object, so the object remains sharp but the
background is blurred instead.

 

Tearing

Tearing is visual distortion during rendering of image (i.e. this distortion can appear only in computer generated graphics, like computer games).

Tearing happens when we try render to the same buffer while it’s being drawn to the monitor, resulting in an old image on the upper half of the screen, with the new image on the lower one (the full clip):