On motion vectors over picture boundaries (or unrestricted MV mode)

History:

The unrestricted motion vector mode appeared already in H.263 video standard (the predecessor of h264/avc, as extension “Unrestricted Motion Vectors”). The idea of Unrestricted Motion Vectors is not new.

To best my recollection the first serious description of Unrestricted Motion Vectors was in the paper “Image and Video Coding—Emerging Standards and Beyond“, by Barry G. Haskell et al., dated 1998.

 

The syntax element motion_vectors_over_pic_boundaries_flag is signaled in SPS extension (VUI) and specifies whether motion vectors can cross the picture boundaries in the given video sequence or not (sometime that feature is called – Unrestricted MV mode).

The H.264/AVC standard permits motion vectors to cross picture boundaries and to point outside the image. One might ask, if a motion vector points to outside of the reference picture (i.e. beyond the previously-decoded frame). So, how reference samples for inter prediction are determined in such case? The answer is simple – the reference frame is extrapolated beyond the image boundaries by repeating the edge samples (e.g. the top line of the picture is duplicated to enable getting full block reference providing that a motion vector points outside the top boundary).

Another question might be also brought up – what’s a gain in coding efficiency to utilize motion vectors that crossing the picture boundaries? It has been found that if a movement occurs close to the frame boundaries than small “jumps” outside the boundary can be optimal:

 

 

 

So, why this flag motion_vectors_over_pic_boundaries_flag is so important to signal it?  Let’s consider one of the methods of parallel encoding of large resolution video (e.g. 4K).

Encoder’s side: We divide each frame into N equal horizontal stripes (slices) and encode each stripe separately (as a sub-frame) on separate processor. We use N processors to encode each stripe in parallel, no hand-shaking between the processors is needed. References of each slice (sub-frame) are collocated slices of the previous frames. We actually generate N separate self-contained video streams, where each stream represents a portion of the high-resolution original frame.

Decoder’s side: we pack back all sliced video streams (in our case vertically) to the original frame. Notice that some manipulation with bits in the slice header is needed: modification of first_mb_in_slice.

If a motion vector in one of sliced stream crosses the top slice boundary then after the packing process this motion vector points to samples of an upper slice and not to the duplicated samples of its own slice, we get a mismatch in references between the encoder and the decoder. Consequently serious visual distortions are observed.

Single sliced stream, decoding is free of visual impairments:

 

                                                                  

 

After stacking two sliced streams, black blocks are observed (encircled by red color).

 

 

Thus, to make sliced stream “vertically concatenable” one has to disable crossing MVs over top and bottom boundaries. If motion_vectors_over_pic_boundaries_flag is set 1 then visual distortion might be appear.

Packing of sliced streams requires modifications of two parameters in slice header: disable_deblocking_filter_idc and first_mb_in_slice . The first must be equal to 2 to disable deblocking across slice boundaries and the second should be updated according to the position of the current slice in a frame.  The modification is easy if entropy mode is CABAC, in such case slice header ends at byte boundary.

The author have a relevant python scripts to pack separate self-contained sliced streams  into compliant video stream (request pls. address to email: sriabtsev@gmail.com).

Fortunately, HW encoder vendors like NVIDIA are familiar with this issue and provides a corresponding support (Constrained Encoding Mode),

for details see www.ffmpeg.org/doxygen/3.4/nvEncodeAPI_8h_source.html:

Indicates Constrained Encoding mode support.

Support added from NvEncodeAPI version 2.0.

     0 : Constrained encoding mode not supported.

      1 : Constarined encoding mode supported.

If this mode is supported then client can enable this during initialisation and disable_deblocking_filter_idc will be set to 2 and prediction vectors for inter macroblocks in each slice will be restricted to the slice region.