DOS on RealSecure 3.2

[Andre Fucs de Miranda <afucs () modulo com br>]

Bulletin #: 243
Title: Denial of Service RealSecure
Information Date: 8/4/00
Product: Realsecure
Company: ISS - Internet Security Systems
Issued by: Módulo Security Labs


Abstract:

The Modulo Security Labs Team found during a test program two ways to stop
the ISS RealSecure 3.2.x engine. The engine is the responsible for the
duty of checking and logging packets. The exploit is very simple to be
reproduced and protection measures must be adopted.

Tested systems:

3.2.1 Solaris - Vulnerable
3.2.2 Solaris - Vulnerable
3.2.1 WinNT - Vulnerable


Solution:

The tests with the Solaris version indicates that disabling the SynFlood
and IPFRAG attacks detection can avoid the 'network_engine' process
failure.

Exploit:

A failure in the treatment of fragmented packets with the SYN flag setted
causes the immediate failure in the RealSecure engine, disabling the
intrusion detection.

On the Solaris version of RealSecure the engine proccess
('network_engine') is disabled, causing a core dump memory file creation.
The event is immediately reported through the RealSecure console.

On the NT version, the engine service file ('network_engine.exe') has a
little different bug. The service, after being crashed, restarts
immediately, generating just a Windows NT Application Log event. The tests
showed that a big and continuous stream of the these packets (SYN Flood)
can take the processor load up to 100%. During this attack, RealSecure
could not identify any other type of attack.

The tests showed that the Solaris version have an additional vulnerability
on the SYN packets treatment. With a SYN Flood attack with specific IP
flags setted it is possible to disable the engine in the same way as
described above. A 50 packets per minute attack was enough to cause the
flaw in a simulation.
On both versions (NT and Solaris) the console could not report the
fragmented attack. The NT version can identify the fragmented SYN attack
as a
simple SYN Flood.

Additional Information:

A detailed version of this advisory will be issued as soon ISS fix the product.

Modulo Security Labs - Modulo Security Solutions
http://www.modulo.com.br/