Outline of Multiplexor

 Basic Assumptions

• Video fps is constant
• Video stream carries PCRs
• PES = frame
• Audio frame contains  1024 samples (e.g. AAC)

DTS of k-th video frame is specified as      

    VDTS(k) = VDTS(0) + k / fps               (1)

where VDTS(0) is initial DTS of the very first video frame (usually DTS(0)=PTS(0)), ‘fps‘ is the frame rate in Hz.

It’s worth mentioning that the frame rate can be variable (e.g. in Cloud Gaming applications video is generated by GPU and the frame rate might be slightly below 60Hz on difficult scenes). Our assumption is that ‘fps’ is fixed  across the whole streaming (i.e. fixed_frame_rate_flag = 1), otherwise the below-described method is incorrect.    

PTS of k-th audio frame is:

    APTS(k) = APTS(0) + k*1024/samplingRate       (2)

  samplingRate can be 44kHz, 48kHz and so on

Each frame of size frameLen is approximately divided into framLen/184 ts-packets. Notice that the first ts-packet in a frame has payload smaller than 184 bytes due to ts-header extension (e.g. to signal PCR) and PES header. Notice: a PES header must always follow the TS header.

We need multiplex two elementary streams: video stream and audio stream.  To prepare these stream to multiplexing we attach VDTS/APTS to  each frame according to the formulae (1) and (2).

 

Multiplexing

Step1:  Packetization of Video Elementary Stream:

    VideoList = []

    For k-th video frame  Do

• • Split the frame into packets, notice the first packet of each frame carries PES header and the last packet probably contains stuffing bytes (the frame size is not necessarily is divided by 184).
  • Attach to each ts-packet a timestamp PT as follows:

        If packet is the first packet in the k-th frame:   PT  = VDTS(k)

        Otherwise :   PT  = PT + 184/fps*frameLen(k)

        Append the packet with its PT to VideoList.

    EndFor

 

 

Step 2: Packetization of Audio Elementary Stream:

   AudioList = []   

     For k-th audio frame  Do

• • Split the frame into packets, notice the first packet carries PES header and  the last packet probably contains stuffing bytes (the frame size is not necessarily is divided by 184).
  • Attach to each ts-packet a timestamp PT as follows:

        If packet is the first packet in the k-th frame :   PT  =  APTS(k)

        Otherwise :   PT  = PT + 1024*184/samplingRate*frameLen(k)

        Append the packet with its PT to AudioList.

    EndFor

So, we have two lists of ts-packets: VideoList and AudioList. The next step is transmission in PT-order.

 

Step3 – Interleaving of ts packets from video and audio lists: 

While either VideoList is non-empty or AudioList is non-empty  Do

• • Look at two ts-packets at the start of Video and Audio lists and pick the packet with the minimal  time (PT) between the two.
  • Put the packet into output Transport Stream

EndWhile

 

Notes:

• • Selection the packet with minimal PT avoids the starvation situation, e.g. when video packets are transmitted while audio packet is waiting a long while.
  • PCR is recommended to be signaled at the start of each video frame (or audio frame) and equals to DTS – PCR_Offset

 

TS Packetization

 

 

General Notes

• Don’t put any media packet before the very first PCR in the stream. For example, if the video stream carries PCRs and you put audio packets before the very first PCR in the stream then some players would not play these audio packets.

• PES Alignment – the first byte of payload in every PES packet begins with the first byte of an access unit. PES alignment is often required.

• At the start of key frame, it’s recommended to put PSI packet (program association tables (PAT), program map tables (PMT),
  network information tables (NIT), and conditional access tables (CAT). The PSI provides control and management information of the stream structure (e.g. elementary stream types and PIDs)