CVE-2016-3623 libtiff: Divide By Zero in the rgb2ycbcr tool

[王梅 <wangmei () 360 cn>]

Details
=======

Product: libtiff
Affected Versions: <= 4.0.6
Vulnerability Type: Divide By Zero
Vendor URL: http://www.remotesensing.org/libtiff/
CVE ID: CVE-2016-3623
Credit: Mei Wang of the Cloud Security Team, Qihoo 360

Introduction
============

Division by zero occurs in rgb2ycbcr in libtiff-4.0.6 allows attackers to cause a denial of service when the param v or 
param h was set to 0.


libtiff-master/libtiff/rgb2ycbcr.c:256-257

250 cvtRaster(TIFF* tif, uint32* raster, uint32 width, uint32 height)
251 {
252         uint32 y;
253         tstrip_t strip = 0;
254         tsize_t cc, acc;
255         unsigned char* buf;
256         uint32 rwidth = roundup(width, horizSubSampling);
257         uint32 rheight = roundup(height, vertSubSampling);
258         uint32 nrows = (rowsperstrip > rheight ? rheight : rowsperstrip);


gdb rgb2ycbcr

(gdb)r -c zip  -r 0  -h 2  -v 0 ./sample/rgb2ycbcr_cvtRaster.tif 1.tif

Program received signal SIGFPE, Arithmetic exception.
0x00000000004017cd in cvtRaster (tif=0x604010, raster=0x7ffff4cab010, width=65312, height=152) at rgb2ycbcr.c:257
257             uint32 rheight = roundup(height, vertSubSampling);
(gdb) p height
$1 = 152
(gdb) p vertSubSampling
$2 = 0

(gdb) r -c zip  -r 0  -h 0  -v 2 ./sample/rgb2ycbcr_cvtRaster.tif 1.tif

Program received signal SIGFPE, Arithmetic exception.
0x0000000000401798 in cvtRaster (tif=0x604010, raster=0x7ffff4cab010, width=65312, height=152) at rgb2ycbcr.c:256
256             uint32 rwidth = roundup(width, horizSubSampling);
(gdb) p width
$3 = 65312
(gdb) p horizSubSampling
$4 = 0

References:
[1] http://www.remotesensing.org/libtiff/
[2] http://bugzilla.maptools.org/buglist.cgi?product=libtiff


Thank you!
Best Regards,


Mei