Get webhook notifications whenever Network & Infrastructure creates an incident, updates an incident, resolves an incident or changes a component status.
We are experiencing a major issue on the vac platform. We are investigating.
Update(s):
Date: 2015-11-19 13:06:09 UTC We've recovered the traces on Vac2 for Cisco.
Vac2 is up again.
Date: 2015-11-19 13:05:43 UTC vac1 and 3 are up.
Date: 2015-11-19 13:05:32 UTC The 3 Vacs have crashed again.
Date: 2015-11-19 13:05:06 UTC Vac1 is up again, the 2 linecards are replaced.
Date: 2015-11-19 13:04:48 UTC We're isolating VAC1 to replace linecards M2.
The protection will be managed by vac2 and 3 during the maintenance.
Date: 2015-11-19 13:04:18 UTC Vac2 is now up to date.
Date: 2015-11-19 13:04:05 UTC We're isolating vac2 to update it.
Date: 2015-11-19 13:03:50 UTC Vac2 is UP.
Date: 2015-11-19 13:03:40 UTC Vac 1 and 3 are online, we keep vac2 off for the troubleshoot with Cisco.
At the moment, it's a hard bug on cards M2.
We doubt that the 3 chassis at the same time is caused by a hardware issue.
However, we launch the RMA for vac1 while keeping troubleshooting.
Date: 2015-11-19 13:01:31 UTC We've reloaded the cards on vac1 and 3, it worked for 20 minutes and then broke down again. We're currently with the TAC Cisco to troobleshoot.
Date: 2015-11-17 03:46:04 UTC The malfunction reoccurred,our teams are on field investigating the situation.
Date: 2015-11-16 15:11:18 UTC We are in the process of troubleshooting with Cisco.
The failing of loopback logs is not the root-cause of default on linecards, but its consequence.
We continue the investigations to determine if the cause is hard or soft
Date: 2015-11-16 15:09:20 UTC At 3:45 GMT+1, we had a simultaneous crash on linecards from 3 vacs RBX, SBG and BHS
2015 Nov 15 05:04:11 admin %DIAG_PORT_LB-2-REWRITE_ENGINE_LOOPBACK_TEST_FAIL: Module:4 Test:RewriteEngine Loopback failed 10 consecutive times. Faulty module:Module 1 Error:Loopback test failed. Pack
ets lost on the SUP in the transmit direction
2015 Nov 15 05:04:11 admin %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on vsh.31048
2015 Nov 15 05:06:31 admin %DIAG_PORT_LB-2-REWRITE_ENGINE_LOOPBACK_TEST_FAIL: Module:3 Test:RewriteEngine Loopback failed 10 consecutive times. Faulty module:Module 1 Error:Loopback test failed. Pack
ets lost on the SUP in the transmit direction
2015 Nov 15 05:06:31 admin %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on vsh.32607
2015 Nov 15 05:07:01 admin %DIAG_PORT_LB-2-REWRITE_ENGINE_LOOPBACK_TEST_FAIL: Module:4 Test:RewriteEngine Loopback failed 10 consecutive times. Faulty module:Module 1 Error:Loopback test failed. Pack
ets lost on the SUP in the transmit direction
2015 Nov 15 05:07:02 admin %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on vsh.468
2015 Nov 15 05:20:43 admin %AUTHPRIV-3-SYSTEM_MSG: pam_aaa:Authentication failed from console - login
2015 Nov 15 05:06:09 admin-vac2 %$ VDC-1 %$ %DIAG_PORT_LB-2-REWRITE_ENGINE_LOOPBACK_TEST_FAIL: Module:4 Test:RewriteEngine Loopback failed 10 consecutive times. Faulty module:Module 1 Error:Loopback test failed. Packets lost on the SUP in the transmit direction
2015 Nov 15 05:06:12 admin-vac2 %$ VDC-1 %$ %DIAG_PORT_LB-2-REWRITE_ENGINE_LOOPBACK_TEST_FAIL: Module:3 Test:RewriteEngine Loopback failed 10 consecutive times. Faulty module:Module 1 Error:Loopback test failed. Packets lost on the SUP in the transmit direction
We have reloaded the linecards.
4:30 GMT+1 , VAC1 was operational.
5:00 GMT+1, the service was totally restored.
We will work with the equipment supplier in order to find the cause of this crash and how it could happen again.