How Generate DTS/PTS from Video Elementary Stream?

Elementary AVC/H.264 stream does not contain timing information (excluding rare cases when presentation times are signaled in picture-timing SEI messages). Therefore, in order to encapsulate elementary stream into mp4 container or mpeg-system program it’s required to compute DTS/PTS (according to the predefined frame rate – fps) such that the following conditions are met:

1) DTS<=PTS

2) differences between PTS magnitudes of successive frames (in play order)  is constant and equal to 1/fps (where fps is frame rate). In other words PTS of current frame differ from PTS of the previous frame (in presentation order) equal to the frame duration (i.e. 1/fps).

3) If HRD is used then PTS-DTS gap should be small in order not to violate DPB buffer.

 

Let’s for simplicity assume that both DTS and PTS are in frame duration units (in case of interlace video – in field duration units) – 1/fps seconds

Case 1: [B frames are not used]

DTS = PTS and DTS is proportional to the frame counter

Case 2:  [B frames are present but not used as reference (i.e. no B-pyramid)]

Let’s assume that GOP structure is fixed in the sense that the interval M between two I/P frames in encoding order is constant, in order words M is equal to the number of B frames between successive I/P frames. For example, M for IPBBPBB… cadence is 2, M for IPBPBIB… cadence is 1 and M for IPPPPPP is 0 respectively

Calculation of DTS/PTS can be outlined as follows:

For the very first frame  DTS = 0 and PTS = 1 (one frame period delay)

For other frames Do

If frame is B then PTS = DTS   (since B frame is not used for reference and there is no sense to delay it in DPB)

If frame is P or I  then  PTS = DTS + M + 1

EndFor

 

If M is not constant (scene cuts can make encoders to break GOP structure) then take M equal to the maximal number of successive B-frames between I/P frames (in encoding order).

For example, for BluRay streams we know a-priory that M does not exceed 3 (BluRay spec. does not allow more than 3 consecutive B frames present).

There is a flaw in taking M as the maximal number of successive B-frames –   we guarantee DTS<=PTS, however differences between PTS magnitudes of successive frames (in play order) might be not equal to the frame duration and consequently the playback might be stuttering.

The following example illustrates PTS stuttering:

Encoding order:

I0  P1  B2   B3   P4   B5   B6  B7   P8   B9

Play order

I0  B2   B3  P1 B5  B6   B7   P4  B9   P8

Take M = 3

Frame       DTS       PTS

I0                0            1

P1              1             5

B2              2            2

B3              3            3

P4              4            8

B5              5            5

B6              6            6

B7             7            7

P8             8            12

B9             9             9

So,  PTS playback is not smooth because difference in PTS between two successive frames (in play order) e.g. B3 and P1 is 2 frame periods and not 1.

To resolve this PTS gap issue, for each I/P frame we have to look-ahead and to determine how many B frames following the current I/P frame. 

If we come back to the above example, for P1 the interval M=2 and hence PTS of P1 equal to 4 instead of 5.