It would be good to hear what caused the problem...
I'm sure many of us will learn from it... Sounds like a nasty :(
From: Simon Blake [mailto:firstname.lastname@example.org]
Sent: Sunday, 7 October 2007 00:56
To: Tony Wicks
Subject: Re: [nznog] Citylink problems?
On Sat, Oct 06, 2007 at 04:40:48PM +1300, Tony Wicks said:
Well Its been more than a couple of more hours now !
A 6-10 hour outage for most, a couple of buildings are still out as of
midnight. Not the best way to spend a sunny Saturday.
There had better be a good explanation
I'll do a more in depth explanation tomorrow. Essentially, this started
happening, on every interswitch trunk on most switches on the network:
Oct 6 14:09:07: %ETHCNTR-3-LOOP_BACK_DETECTED: Keepalive packet loop-back
detected on FastEthernet0/24.
Oct 6 14:09:07: %PM-4-ERR_DISABLE: loopback error detected on Fa0/24,
putting Fa0/24 in err-disable state Oct 6 14:09:08: %LINEPROTO-5-UPDOWN:
Line protocol on Interface FastEthernet0/24, changed state to down Oct 6
14:09:09: %LINK-3-UPDOWN: Interface FastEthernet0/24, changed state to down
Oct 6 14:09:50: %PM-4-ERR_RECOVER: Attempting to recover from loopback
err-disable state on Fa0/24 Oct 6 14:09:55: %LINK-3-UPDOWN: Interface
FastEthernet0/24, changed state to up Oct 6 14:09:57: %LINEPROTO-5-UPDOWN:
Line protocol on Interface FastEthernet0/24, changed state to up Oct 6
14:09:57: %ETHCNTR-3-LOOP_BACK_DETECTED: Keepalive packet loop-back detected
a problem that takes a while to resolve, when switches are only reachable
for 5 of every 45 seconds. It wasn't a fibre cut, it wasn't a power outage,
it wasn't a hardware failure, it wasn't obviously a spanning-tree wigout, we
had no scheduled (or unscheduled) work on at that time.
and I really love the -
Network Status - No known fault
System Status Normal - No major problems
Priorities. When the network is so hosed that I can't see the website from
the other end of the DMZ, making sure the website is uptodate takes a back
seat to actually fixing the network. Once we got Internet connectivity back
to our office, we got the page updated pretty smartly
- just before 5pm, from memory.
NZNOG mailing list