ffmpeg audio and video out of sync solution
After research, it was found that there are two places to control the timestamp for writing when the FFMPEG SDK writes the video, one is AvPacket and the other is AvFrame. When calling avcodec_encode_video, you need to pass in the object pointer of AvFrame, which is to pass in a frame of uncompressed video for compression processing. AvFrame contains a pts parameter, which is the timestamp of the current frame when it will be played in the future. And AvPacket also has pts and dts. Speaking of which, it is necessary to explain the I, P, and B video compression frames. The I frame is the key frame and does not depend on other video frames. The P frame is a forward predicted frame and only depends on the previous video frame. The B frame is a bidirectional predicted video frame and depends on the front and rear video frames. Because of the existence of the B frame, because it is bidirectional, you must know the details of the previous video frame and the following video frame to know what image should be presented in the B frame. The two parameters pts and dts are used to control the display and decoding order of video frames.
pts is the order in which frames are displayed.
dts is the order in which frames are read and decoded.
If no B frame exists, dts and pts are the same. On the contrary, it is different. For a detailed introduction to this, please refer to the principle of mpeg.
Let's talk about what values ​​should be set for the two pts and dts included in AvPacket?
What pts and dts need to set is the order of video frame decoding and display. One is added for every additional frame, not the timestamp of playing the video.
But practice has proved that the video decoded by rmvb is sometimes not a fixed frame rate, but a variable frame rate. In this way, if each frame is compressed, the pts and dts plus one solution will cause the audio and video to be out of sync.
Solve the audio and video out of sync method:lTImeStamp is the timestamp of the current video frame obtained through directshow.
m_llframe_index is the number of frames that have currently undergone compression processing.
First, av_rescale calculates what timestamp video frame the current compression process has to process. If the timestamp has not reached the timestamp of the video frame currently provided by directshow, the frame is discarded.
Otherwise, the compression operation is performed. And set the pts and dts of AVPacket. It is assumed here that the B frame does not exist.
Because the video will be played at the fixed playback frame rate we set in the future, it is necessary to compare the video frame time stamp calculated according to the set playback frame rate with the current video frame time stamp provided by directshow. Do you need to implement a strategy to delay playback? If you need to delay playback, increase the pts by a step of 2, otherwise play at normal speed, then set to 1.dts to be the same.
__int64 x = av_rescale (m_llframe_index, AV_TIME_BASE * (int64_t) c- "TIme_base.num, c-" TIme_base.den);
if (x "lTimeStamp)
{
return TRUE;
}
m_pVideoFrame2- 《pts = lTimeStamp;
m_pVideoFrame2- 《pict_type = 0;
int out_size = avcodec_encode_video (c, m_pvideo_outbuf, video_outbuf_size, m_pVideoFrame2);
/ * if zero size, it means the image was buffered * /
if (out_size << 0)
{
AVPacket pkt;
av_init_packet (& pkt);
if (x "lTimeStamp)
{
pkt.pts = pkt.dts = m_llframe_index;
pkt.duration = 0;
}
else
{
pkt.duration = (lTimeStamp-x) * c- 《time_base.den / 1000000 + 1;
pkt.pts = m_llframe_index;
pkt.dts = pkt.pts;
m_llframe_index + = pkt.duration;
}
//pkt.pts = lTimeStamp * (__int64) frame_rate.den / 1000;
if (c- 《coded_frame && c- 《coded_frame- 《key_frame）
{
pkt.flags | = PKT_FLAG_KEY;
}
pkt.stream_index = m_pVideoStream- 《index;
pkt.data = m_pvideo_outbuf;
pkt.size = out_size;
/ * write the compressed frame in the media file * /
ret = av_interleaved_write_frame (m_pAvFormatContext, & pkt);
}
else
{
ret = 0;
}
7/8" mini female angle connectors used for power support .The ingress protection is available and rated to IP 67, these connectors are ideally suited for industrial control networks where small Sensors are required. Connectors are either factory TPU over-molded or panel receptacles supplied with sold-cup for wire connecting or with PCB panel solder contacts. Field attachable / mountable Connector is also available for your choice.
Mini Male Connector,7/8" Connector,Field Wireable Male Connector,Mini Connector 5 Pin
Kunshan SVL Electric Co.,Ltd , https://www.svlelectric.com