CVE-2017-14160: libvorbis-1.3.5 bark_noise_hybridmp() integer signedness bug

[连一汉 <lianyihan () 360 cn>]

Hi,

I’m a security researcher of Qihoo 360 GearTeam.
My partner Zhibin Hu and I found a vulnerability of libvorbis-1.3.5.
And we have applied for CVE-2017-14160 of this vulnerability.
================== test command ====================

ffmpeg �Ci poc.mp4 �Cy 1.mkv
// libvorbis-1.3.5 has been compiled into ffmpeg static.

================= needed version ====================

I compile it as https://github.com/google/oss-fuzz/blob/master/projects/ffmpeg/build.sh

This is the problem of libvorbis-1.3.5, and I tried libvorbis in ubuntu repo, it could also trigger this vul or bug.

=================== crash info ======================

(gdb) bt
#0  0x0000000001f95afd in bark_noise_hybridmp (n=256, b=0x32cd940, f=0x32e5010, noise=0x32f7ed0, offset=140, fixed=-1) 
at psy.c:630

#1  0x0000000001f95430 in _vp_noisemask (p=0x32aa820, logmdct=0x32e5010, logmask=0x32f7ed0) at psy.c:705
#2  0x0000000001facac9 in mapping0_forward (vb=0x329cfb0) at mapping0.c:417
#3  0x0000000001f92c9e in vorbis_analysis (vb=0x329cfb0, op=0x0) at analysis.c:46
#4  0x0000000000bc2725 in libvorbis_encode_frame (avctx=0x329ca00, avpkt=0x32ab540, frame=0x32e4400, 
got_packet_ptr=0x7fffffffdbf4) at libavcodec/libvorbisenc.c:311
#5  0x00000000009e5717 in avcodec_encode_audio2 (avctx=0x329ca00, avpkt=0x32ab540, frame=0x32e4400, 
got_packet_ptr=0x7fffffffdbf4)at libavcodec/encode.c:198
#6  0x00000000009e62d8 in do_encode (avctx=0x329ca00, frame=0x32e4400, got_packet=0x7fffffffdbf4) at 
libavcodec/encode.c:375

#7  0x00000000009e6224 in avcodec_send_frame (avctx=0x329ca00, frame=0x32e4400) at libavcodec/encode.c:421
#8  0x0000000000438ef5 in do_audio_out (of=0x3299560, ost=0x329c7a0, frame=0x32e4400) at ffmpeg.c:921
#9  0x0000000000436c5b in reap_filters (flush=0) at ffmpeg.c:1515
#10 0x000000000042dc30 in transcode_step () at ffmpeg.c:4553
#11 0x000000000042bc49 in transcode () at ffmpeg.c:4597
#12 0x000000000042b092 in main (argc=5, argv=0x7fffffffe678) at ffmpeg.c:4803

(gdb) l
625
626         lo = b[i] >> 16;
627         hi = b[i] & 0xffff;
628         if(hi>=n)break;
629
630         tN = N[hi] - N[lo];
631         tX = X[hi] - X[lo];
632         tXX = XX[hi] - XX[lo];
633         tY = Y[hi] - Y[lo];
634         tXY = XY[hi] - XY[lo];
(gdb) p hi
$4 = 0
(gdb) p lo
$5 = 49656                                                                 // 
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
(gdb) p i
$6 = 259