Re: cisco 7200 performance issue

[Wendy Garvin <wgarvin () cisco com>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Hi Luciano,

High CPU can be caused by a high volume of traffic destined to the processor
on your 7200. The "%RCMD-4-RSHPORTATTEMPT:" message is commonly seen if you
are getting scanned by Nessus, or an nmap based scanner. Your CPU
utilization is not necessarily related to the rsh attempt specifically, but
it could be related to the scanning activity as a whole.

To narrow this down further, you can put up an input access list on all your
interfaces with permit statements for specific types of traffic, and monitor
the 'hit' counts with the 'show access-lists' command. You may also try
specifically blocking rsh traffic just to verify that it's not the rsh
connection attempts that are causing the high CPU.

WARNING! Be careful not to cut off your remote access when applying an input
access-list.

An example input access list on a router with a interface address of 11.0.0.3
might be:
- - --
!!These will help you characterize the scan
access-list 101 permit tcp any host 11.0.0.3 eq cmd
access-list 101 permit tcp any host 11.0.0.3 eq login
access-list 101 permit tcp any host 11.0.0.3 eq nntp
access-list 101 permit tcp any host 11.0.0.3 eq lpd
access-list 101 permit tcp any host 11.0.0.3 eq finger
access-list 101 permit tcp any host 11.0.0.3 eq echo
access-list 101 permit tcp any host 11.0.0.3 eq chargen log-input
!! These will help you identify what services you are running
access-list 101 permit tcp any host 11.0.0.3 eq ftp
access-list 101 permit tcp any host 11.0.0.3 eq telnet
access-list 101 permit tcp any host 11.0.0.3 eq bgp
access-list 101 permit ospf any host 11.0.0.3 
access-list 101 permit ospf any host 224.0.0.6
access-list 101 permit eigrp any host 224.0.0.10
access-list 101 permit eigrp any host 11.0.0.3
access-list 101 permit udp any host 11.0.0.3 eq snmp
access-list 101 permit udp any host 11.0.0.3 eq ntp
!!This will not allow you to identify all critical services
!!Some you may still need to look for are icmp ttl exceeded
!!and ftp/tftp for upgrades.
access-list 101 permit ip any any


interface fastethernet0/0
ip access-group 101 in
- - --

The output of 'sh access-list 101' after a default namp scan might be:

Router#show access-list 101
Extended IP access list 101
    permit tcp any host 11.0.0.3 eq cmd (3 matches)
    permit tcp any host 11.0.0.3 eq login (3 matches)
    permit tcp any host 11.0.0.3 eq nntp (3 matches)
    permit tcp any host 11.0.0.3 eq lpd (3 matches)
    permit tcp any host 11.0.0.3 eq finger (9 matches)
    permit tcp any host 11.0.0.3 eq echo (10 matches)
    permit tcp any host 11.0.0.3 eq chargen log-input (2 matches)
    permit tcp any host 11.0.0.3 eq ftp (5 matches)
    permit tcp any host 11.0.0.3 eq telnet (10 matches)
    permit tcp any host 11.0.0.3 eq bgp (3 matches)
    permit ospf any host 11.0.0.3
    permit ospf any host 224.0.0.6
    permit eigrp any host 224.0.0.10
    permit eigrp any host 11.0.0.3
    permit udp any host 11.0.0.3 eq snmp
    permit udp any host 11.0.0.3 eq ntp
    permit ip any any (4867 matches)

And in your log, you may see this:

Dec  6 16:21:51.350: %SEC-6-IPACCESSLOGP: list 101 permitted tcp
11.0.0.19(35265) (FastEthernet3/0 0002.559c.f66e) -> 11.0.0.3(19), 1 packet

I deliberately chose 'chargen' as a service you aren't likely to have
legitimate connections too, as 'log-input' can cause further increases in
your CPU utilization. This gives you the source/destination of this packet.

If you don't get immediate results, try replacing the address 11.0.0.3 in
your access-list with your loopback address, and try placing the access-list
on various input interfaces until you track down the source of the scan. The
command 'show tcp brief all' may also be useful to you.

At this point you can be fairly certain someone is scanning hosts on your
network. It would be useful to know whether they're scanning your routers
specifically or if they're hitting every host that answers. You may be able
to track the source of the scan by following the traffic from the input
interface back to the next hop router.

If you continue to have high CPU utilization on your router, you may
consider placing input access-lists on your interfaces which protect your
router from traffic destined to the processor. The above access-list should
help you recognize critical services destined to your router, now you must
write an access list to permit only those from legitimate sources, and block
everything else destined to the router.

For example, if you are using ssh to connect to your routers from
11.0.0.0/24, and the only other legitimate service you need to connect to
the router is BGP from a directly connected peer, you could apply the
following access-list:

access-list 101 permit tcp host 11.0.0.2 host 11.0.0.3 eq bgp
access-list 101 permit tcp  11.0.0.0 0.0.0.255 host 11.0.0.3 eq 22
access-list 101 deny ip any any

Please carefully review the traffic currently configured to connect to your
router before applying such an access-list, you may disconnect critical
services or block yourself from managing the router if you do not plan the
access-list correctly.

Please feel free to contact psirt () cisco com for any security concerns you
have with Cisco products. You may also open up a case with our Technical
Assistance Center by sending email to tac () cisco com

Thanks,

- --Wendy

> Luciano Z <user_luciano () yahoo com br> [2003-05-21 16:45] wrote:
>
> Hi!
>
> I was responding an incident last night and saw a
> strange performance problem with a cisco 7200.
>
> When I issued a "sh interface" on the two fast
> ethernets of my box it was show that I got only 6Mbps
> traffic and normal packet per second rate but when I
> "sh logg" the box I got a lot of
> "%RCMD-4-RSHPORTATTEMPT: Attempted to connect to
> RSHELL from x.y.z.w" messages with spoofed sources.
>
> Investigating a little more I discovered that this
> traffic was pushing the CPU to 98% to 100% of
> utilization. Back to the output of "sh logg" I saw
> that the box was logging 2 to 3 RSHELL messages per
> second. In my opinion this coulnd?t affect the CPU so
> much. The router have 256M of RAM and it?s a 7200!
>
> I coulnd?t gather more info about this incident
> because it stopped before I could get the data. The
> strange thing it?s that the high CPU utilization
> stopped too.
>
> I don?t know if this is a problem of this cisco model
> or if I?m missing something. Any ideias?
>
> []
> lwulff
>
> _______________________________________________________________________
> Yahoo! Mail
> O melhor e-mail gratuito da internet: 6MB de espa?o, antiv?rus, acesso POP3, filtro contra spam. 
> http://br.mail.yahoo.com/
>
> ----------------------------------------------------------------------------
> *** Wireless LAN Policies for Security & Management - NEW White Paper ***
> Just like wired networks, wireless LANs require network security policies 
> that are enforced to protect WLANs from known vulnerabilities and threats. 
> Learn to design, implement and enforce WLAN security policies to lockdown enterprise WLANs.
>
> To get your FREE white paper visit us at:    
> http://www.securityfocus.com/AirDefense-incidents
> ----------------------------------------------------------------------------

- - -- 
Wendy Garvin - Cisco PSIRT - 408 525-1888 CCIE# 6526
- - ----------------------------------------------------
           http://www.cisco.com/go/psirt


-----BEGIN PGP SIGNATURE-----
Version: PGP 6.5.2

iQA/AwUBPs3b0M/6vhuARK9tEQLCwQCgwWSjs38+slcpABQLutgKXjErkJ0AniQr
chJvuGNR13SLLnuBI0rjr2dX
=2Nyl
-----END PGP SIGNATURE-----

----------------------------------------------------------------------------
*** Wireless LAN Policies for Security & Management - NEW White Paper ***
Just like wired networks, wireless LANs require network security policies 
that are enforced to protect WLANs from known vulnerabilities and threats. 
Learn to design, implement and enforce WLAN security policies to lockdown enterprise WLANs.

To get your FREE white paper visit us at:    
http://www.securityfocus.com/AirDefense-incidents
----------------------------------------------------------------------------