[x264-devel] x264 seg fault issue when encode 4K video

[홍석진]

Hi.
​
I have a problem when I try to encode some 4K videos with libx264.
​
I'm using x264 options as below
-c:v libx264 -profile:v high -x264opts bframes=2:b_adapt=2:rc_lookahead=48:keyint=60:keyint-min=30 -threads 21 -pix_fmt yuv420p -vf scale=3840:2160 -f mp4.​
​
Actually, I tried to encode by changing arguments as below
- change number of threads (21 -> 16 or 10)
- add force_key_frames - delete bframe options
- change b_adapt (2 -> 1, it means that X264_B_ADAPT_TRELLIS to X264_B_ADAPT_FAST as you know)
​
Changing b_adapt option 2 to 1 works well.
Otherwise sometimes it occurs same seg fault issues.
​
I attached back trace info
#0 0x0000000001f5a721 in x264_slicetype_frame_cost () #1 0x0000000001f5c89e in x264_slicetype_analyse () #2 0x0000000001ef9be6 in x264_stack_align () #3 0x0000000001ef201c in x264_lookahead_slicetype_decide () #4 0x0000000001ef21d9 in x264_lookahead_thread () #5 0x00007ffff72a6e25 in start_thread () from /usr/lib64/libpthread.so.0 #6 0x00007ffff660034d in clone () from /usr/lib64/libc.so.6​
I tried to analyze x264 codes and found something weird situation.
​
​
static int slicetype_frame_cost( x264_t *h, x264_mb_analysis_t *a, x264_frame_t **frames, int p0, int p1, int b ) /* For each list, check to see whether we have lowres motion-searched this reference frame before. */ x264_log( h, X264_LOG_INFO, "[slicetype_frame_cost] p0: %d, p1: %d, b: %d, b-p0-1: %d, p1-b-1: %d\n", p0, p1, b, b-p0-1, p1-b-1); do_search[0] = b != p0 && fenc->lowres_mvs[0][b-p0-1][0][0] == 0x7FFF; do_search[1] = b != p1 && fenc->lowres_mvs[1][p1-b-1][0][0] == 0x7FFF;​
I think this problem is array index out of range exception in x264_slicetype_frame_cost function.
Seg fault occurs when function parameters b=48, p0=44 and p1=48.
It means [b-p0-1] is 3. But fenc->lowres_mvs[0][3] is not allocated.
​
​
cur_nonb = i; // before segmentation fault, i is 47. (i = number of frames in frame buffer - 1) while( IS_X264_TYPE_B( frames[cur_nonb]->i_type ) && cur_nonb > 0 ) cur_nonb--;​
I understood p0 means the index of non-bframe (reverse position) in frames, macroblock_tree function.
​
Before seg fault, frame types of frames changed 1530000000000000000000000000000000000000000000000 to 153553553553553553553553553553553553355333533555.
So that, cur_nonb is calculated 44. After that b-p0-1 be 3.
Except in this case, b-p0-1 and p1-b-1 are always calculated -1 to 2.
​
When I changed frames->i_type X264_TYPE_B to X264_TYPE_P forced if greater than 2 consecutive b frames are exist, it solved.
But I'm not sure whether can be used or not because this solution is not cool.
(Anyway, because of this issue I changed b-adapt option 2 to 1)
​
​
To sum up, seg fault occurs when encode some 4K contents.
By changing x264 options, it can be avoided (especially b-adapt option).
​
Please let me know if you need more information.
​
Regards.

[BugMaster]

Post by 홍석진
Hi.
I have a problem when I try to encode some 4K videos with libx264.
I'm using x264 options as below
-c:v libx264 -profile:v high -x264opts bframes=2:b_adapt=2:rc_lookahead=48:keyint=60:keyint-min=30 -threads 21 -pix_fmt yuv420p -vf scale=3840:2160 -f mp4.?
Actually, I tried to encode by changing arguments as below
- change number of threads (21 -> 16 or 10)
- add force_key_frames - delete bframe options
- change b_adapt (2 -> 1, it means that X264_B_ADAPT_TRELLIS to X264_B_ADAPT_FAST as you know)
Changing b_adapt option 2 to 1 works well.
Otherwise sometimes it occurs same seg fault issues.
I attached back trace info
#0 0x0000000001f5a721 in x264_slicetype_frame_cost ()
#1 0x0000000001f5c89e in x264_slicetype_analyse ()
#2 0x0000000001ef9be6 in x264_stack_align ()
#3 0x0000000001ef201c in x264_lookahead_slicetype_decide ()
#4 0x0000000001ef21d9 in x264_lookahead_thread ()
#5 0x00007ffff72a6e25 in start_thread () from /usr/lib64/libpthread.so.0
#6 0x00007ffff660034d in clone () from /usr/lib64/libc.so.6?
I tried to analyze x264 codes and found something weird situation.
static int slicetype_frame_cost( x264_t *h, x264_mb_analysis_t *a,
x264_frame_t **frames, int p0, int p1, int b )
/* For each list, check to see whether we have lowres motion-searched this reference frame before. */
x264_log( h, X264_LOG_INFO, "[slicetype_frame_cost] p0: %d, p1: %d, b: %d, b-p0-1: %d, p1-b-1: %d\n", p0, p1, b, b-p0-1, p1-b-1);
do_search[0] = b != p0 && fenc->lowres_mvs[0][b-p0-1][0][0] == 0x7FFF;
do_search[1] = b != p1 && fenc->lowres_mvs[1][p1-b-1][0][0] == 0x7FFF;?
I think this problem is array index out of range exception in x264_slicetype_frame_cost function.
Seg fault occurs when function parameters b=48, p0=44 and p1=48.
It means [b-p0-1] is 3. But fenc->lowres_mvs[0][3] is not allocated.
cur_nonb = i; // before segmentation fault, i is 47. (i = number of frames in frame buffer - 1)
while( IS_X264_TYPE_B( frames[cur_nonb]->i_type ) && cur_nonb > 0 )
cur_nonb--;?
I understood p0 means the index of non-bframe (reverse position) in frames, macroblock_tree function.
Before seg fault, frame types of frames changed 1530000000000000000000000000000000000000000000000 to 153553553553553553553553553553553553355333533555.
So that, cur_nonb is calculated 44. After that b-p0-1 be 3.
Except in this case, b-p0-1 and p1-b-1 are always calculated -1 to 2.
When I changed frames->i_type X264_TYPE_B to X264_TYPE_P forced if greater than 2 consecutive b frames are exist, it solved.
But I'm not sure whether can be used or not because this solution is not cool.
(Anyway, because of this issue I changed b-adapt option 2 to 1)
To sum up, seg fault occurs when encode some 4K contents.
By changing x264 options, it can be avoided (especially b-adapt option).
Please let me know if you need more information.
Regards.

Hi.

Do you force frame types in you app using libx264? Because it looks
like bug is cased by 3 B-frames sequence when only 2 B-frames was
allowed (bframes=2 option).

Also can you try attached patch to see if it fix your seg fault? That
is only possible that can in theory cause such problems at 4K content.

If it doesn't help than it would be good to see h->param.i_bframe,
frm->i_type and frm->i_forced_type where `frm` is elements
of frames[0..num_frames-1].

[홍석진]

Hi.
​
Thanks to rapidly support.
I applied patch you sent to me to latest x264 branch and tried to encode few 4K contents which make seg fault problem.
And I finally got normal encoded videos.
​
When I saw patch, it changes variable type of slicetype_path_cost, int(32) to uint64.
It means that when calculate slicetype_path_cost at 4K video contents, this may occurs overflow, right?
But I wonder why it was encoded properly when I changed the number of threads.
​
Do I apply your patch onto latest x264 and can I use it?
I wish you commit this patch to x264 branch because I use pure libx264, without any changes.
​
Once again, thank you for your support.
​
Regards.

[BugMaster]

_______________________________________________
x264-devel mailing list
x264-***@videolan.org
https://mailman.videolan.org/listinfo/x264-devel