nanog mailing list archives

Strange Cisco 6503 problem


From: Dean Belev <dbelev () gmail com>
Date: Thu, 28 Jan 2010 14:56:19 +0200

Hi all,

I experienced a strange problem with one of our Cisco 6503 routers - right after the terminal PC connected to the router via console is rebooted the router reboots itself. Even when there is no Eth connection to the PC the situations is the same - reboot follows.

I tried to check the messages the router sends:

*: %CPU_MONITOR-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 31 seconds [1/0]
:: No EOBC input, dump debuginfo, interval 10, times 3
: %CPU_MONITOR-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 60 seconds [1/0] : %CPU_MONITOR-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 90 seconds [1/0]
: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 1/0: IPC Failure: timeout
: %CPU_MONITOR-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 120 seconds [1/0] : %CPU_MONITOR-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 150 seconds [1/0]*

All the possible reasons according to Cisco that may caused that case will be checked soon because the router is in operational mode although some of them looks strange:

/*CPU_MONITOR-3-TIMED_OUT or CPU_MONITOR-6-NOT_HEARD
Problem

The switch reports these error messages:
CPU_MONITOR-3-TIMED_OUT: CPU monitor messages have failed, resetting system CPU_MONITOR-6-NOT_HEARD: CPU monitor messages have not been heard for [dec] seconds
Description
These messages indicate that CPU monitor messages have not been heard for a significant amount of time. A time-out most probably occurs,
which resets the system.
[dec] is the number of seconds.
The problem possibly occurs because of these reasons:
    *      Badly seated line card or module
    *      Bad ASIC or bad backplane
    *      Software bugs
    *      Parity error
    *      High traffic in the Ethernet out of band channel (EOBC) channel
The EOBC channel is a half duplex channel that services many other functions, which includes Simple Network Management Protocol (SNMP) traffic and packets that are destined to the switch. If the EOBC channel is full of messages because of a storm of SNMP traffic, then the channel is subjected to collisions. When this happens, EOBC is possibly not able to carry IPC messages.
    This makes the switch display the error message.

    Workaround

Reseat the line card or module. If a maintenance window can be scheduled, reset the switch in order to clear any transient issues.
#########################################
Error Message
%ERR_DET-5-ERR_DET_NO_EOBC_INPUT: No EOBC input, dump debuginfo, interval %u, times %u Explanation No Ethernet Out-of-Band Channel (EOBC) input was received. Debugging information will be dumped.
Recommended Action    No action is required.*/

I'm curious if some of you faced such a problem - reboot of the router caused by the console connection.

Here is some brief info about the router:

*Supervisor:*
Mod Ports Card Type                                               Model
--- ----- -------------------------------------- ------------------ -----------
  1    9  Supervisor Engine 32 8GE (Active)      WS-SUP32-GE-3B

*Processor:*
cisco WS-C6503-E (R7000) processor

*IOS:*
IOS (tm) s3223_rp Software (s3223_rp-ADVENTERPRISEK9_WAN-M), Version 12.2(18)SXF8, RELEASE SOFTWARE (fc2)

Thank you in advance for all your replays.

Best~


Current thread: