Linx has now resolved the issues and we have therefore restored our BGP sessions back to the Lon1 Exchange.
Posted Mar 25, 2021 - 23:30 GMT
Update
Linx continues to work on the issue and most providers still have their sessions shutdown including us. However this is causing some odd routing issues for many providers. BT network for example are not able to reach some banking websites / networks.
So for example, Your Internet Provider may have routed traffic to Barclays Bank via London Linx 1, this is shutdown so traffic routes via BT and then fails. Suspect their maybe wider issues with some networks due to the above.
** Response from the London Internet Exchange ** Juniper JTAC have made solid progress throughout the last 12 hours, they were able to recreate the corruption issue during the UK night. Though their L2ALD Software Engineering Experts are continuing to investigate the root cause. They will continue their investigations for another few hours before we proceed with our interim working model as detailed in our last update.
The network remains stable and we will continue to hold off on any provisioning work for the time being. We have a meeting with Juniper at 13:00 GMT and will update via ops-announce soon after.
** Response from the London Internet Exchange **
Until London Internet Exchange confims the issue is fullly resolved, most providers including us, will leave sessions shutdown to prevent outages in the event of the London Internet Exchange having issues again.
Posted Mar 25, 2021 - 09:25 GMT
Update
Just a quick update. The London Internet Exchange are stil working on the issue and our BGP Sessions remain shutdown to protect customers from issues.
If anyone is unable to reach our network or are having issues with VoIP Services connecting, please let us know.
Posted Mar 24, 2021 - 12:17 GMT
Identified
We have had the below statement from the London Internet Exchange.
Our BGP Sessions wih Linx's remain shutdown
** From London Internet Exchange **
LINX LON1 instability issues 23.03.21
At around 11:15 GMT and through to 11:28 GMT, whilst debugging a member issue we saw L2 forwarding disruption on the edge2-thn device impacting a small number of members directly connected.
Furthermore, at around 11:41-11:57 GMT a provisioning change pushed to the edge device edge1-thw appeared to cause a similar L2 forwarding issue.
In both these cases we escalated to the Juniper TAC centre and we decided to halt any further provisioning changes to the LON1 network.
At 15:32 GMT we started to see issues after adding debugging commands as requested by TAC and whilst the L2 issues were quickly mitigated this did cause significant disruption to traffic.
Since the last LON1 disruption at approx. 15:40 GMT the network has remained stable. As a result of adding the debugging we are able to collect more data for the Juniper TAC who are still in the process of investigating this issue. Currently we know a trigger of the issue is any new commits on devices made can affect the L2 forwarding of the device. For this reason we will continue with the suspension of provisioning new services on LON1.
If any members decide to withdraw their peering from LON1 for the time being can you please shutdown your peering and not your ports.
We will provide further updates throughout the evening and to the incident conclusion.
** Freom London Internet Exchange **
Posted Mar 23, 2021 - 17:47 GMT
Update
We are now shutting down our session to Linx Lon1 to prevent conenctions from flapping and to ensure traffic is routed away from the exchange.
Posted Mar 23, 2021 - 15:49 GMT
Investigating
London Internet Exchange Lon1 is currently down. This is one of the UKs Main Peering Hub's where most ISPs will connect to for interconenctions between each other.
We are currently chasing Linx's for an update. meantile traffic will take anoher path to reach our network.
Posted Mar 23, 2021 - 15:47 GMT
This incident affected: Network Infrastructure (London LINX Lon 1).