Postmortem -
Read details
Dec 4, 11:38 GMT
Resolved -
last Router now backup and issue should now be fully resolved.
Dec 4, 04:57 GMT
Update -
The new software is being loaded onto the second router now and will require a few reboots.
Dec 4, 04:32 GMT
Update -
The second router is also stable at the moment, However due to the BGP DDOS, we are going to do the upgrade now.
We shall be starting the work shortly after 4:30am,
Dec 4, 03:58 GMT
Update -
Router is now stable.
Dec 4, 03:50 GMT
Update -
Juniper Networks Junos OS upgrade finished and doing final reboots on 1st Router.
Second Router can be upgraded from within the network so no need for Phil to go to second Data Centre.
Network at Volta should start to become stable again very shortly.
Dec 4, 03:29 GMT
Update -
Phil has now arrived onsite and will proceed to upgrading the Router Software, once updated and router is stable, Phil will attend second data centre and upgrade the software on the other Juniper MX Router.
Update to follow.
Dec 4, 01:53 GMT
Update -
Phil is expected to arrive on-site at 1:39am
Dec 4, 01:24 GMT
Identified -
Possible Cause of issue
An Improper Handling of Exceptional Conditions vulnerability in the routing protocol daemon (RPD) of Juniper Networks Junos OS
When a malformed BGP UPDATE packet is received over an established BGP session, RPD crashes and restarts.
Continuous receipt of the malformed BGP UPDATE messages will create a sustained Denial of Service (DoS) condition for impacted devices.
It would seem someone remotely is trying to cause a Denial of Service (DoS) attack on part of our network.
Phil is now on his way to Sov House to load a software update on the affected router to hopefully resolve the issue.
Dec 4, 01:14 GMT
Update -
We continue to investigate and are taking steps to resolve the issue.
Dec 4, 00:47 GMT
Update -
The Router has rebooted, however the BGP sessions are still having issues.
We are investigating to find out the cause.
Dec 4, 00:34 GMT
Investigating -
We are currently aware of a issue with BGP Flapping which is causing parts of our network to be unreachable.
We are rebooting the affected Router to clear the fault.
Dec 4, 00:27 GMT