tag:status.it-communicationsltd.co.uk,2005:/historyI.T Communications Limited Status - Incident History2024-03-29T07:51:46+00:00I.T Communications Limitedtag:status.it-communicationsltd.co.uk,2005:Incident/202643382024-04-17T20:00:00+01:002024-03-15T11:01:01+00:00Essential Network Maintenance CHG0090504<p><strong>THIS IS A SCHEDULED EVENT Apr <var data-var='date'>17</var>, <var data-var='time'>20:00</var> BST - Apr <var data-var='date'>18</var>, <var data-var='time'>05:03</var> BST</strong></p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>11:00</var> GMT</small><br><strong>Scheduled</strong> - Please be aware we need to carry out essential maintenance which will affect the service we provide you as follows:<br /><br />Maintenance Ref CHG0090504<br /><br />Planned Start Date: 17/04/2024 Planned Start Time: 20:00<br />Planned End Date: 18/04/2024 Planned End Time: 06:00<br />Outage Start Date: 17/04/2024 Outage Start Time: 22:30<br />Outage End Date: 18/04/2024 Outage End Time: 06:00<br /><br /><br />Due to our redundant network, Customers will not see any impact to service. This will mean that our Data Centre site links will have a reduced capacity during the maintenance from 200Gb down to 100Gb.</p>tag:status.it-communicationsltd.co.uk,2005:Incident/202642502024-04-09T22:30:00+01:002024-03-15T10:57:01+00:00Essential Network Maintenance CHG0090466<p><strong>THIS IS A SCHEDULED EVENT Apr <var data-var='date'>9</var>, <var data-var='time'>22:30</var> BST - Apr <var data-var='date'>10</var>, <var data-var='time'>06:00</var> BST</strong></p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>10:57</var> GMT</small><br><strong>Scheduled</strong> - Please be aware we need to carry out essential maintenance which will affect the service we provide you as follows:<br /><br />Maintenance Ref CHG0090466<br /><br /><br />Planned Start Date: 09/04/2024 Planned Start Time: 22:30<br />Planned End Date: 10/04/2024 Planned End Time: 06:00<br /><br />Outage Start Date: 09/04/2024 Outage Start Time: 22:30<br />Outage End Date: 10/04/2024 Outage End Time: 06:00<br /><br />Due to our redundant network, Customers will not see any impact to service. This will mean that our Data Centre site links will have a reduced capacity during the maintenance from 200Gb down to 100Gb.</p>tag:status.it-communicationsltd.co.uk,2005:Incident/199001892024-02-03T15:50:07+00:002024-02-06T18:53:59+00:00Service Issue<p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>15:50</var> GMT</small><br><strong>Resolved</strong> - Power fully restored, San Storage Restored and all servers are now back online.<br /><br />We will follow up on Monday with the Data Centre to understand the reason for the power failure.<br /><br />We have been located within the Volta Data Centre since Feb 2016 and never had a issue like this before, so it is rare.<br /><br />We are sorry for the outage today. Full RFO will be published once received from the Data Centre.</p><p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>14:43</var> GMT</small><br><strong>Update</strong> - We have now engaged our San Storage provider (NETAPP) due to a issue caused by the power failure, Preventing the storage service from starting. <br /><br />A NetAPP engineer has now been assigned of which they are now working with us to resolve the issue.</p><p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>13:59</var> GMT</small><br><strong>Update</strong> - We now have power restored. We are waiting for our San Storage to come back online. Once the San is fully working, we will start up all the servers which will then restore the service.</p><p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>13:09</var> GMT</small><br><strong>Update</strong> - We do not have an ETR at this time. The DC currently have two UPS systems offline due to a controls issue.</p><p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>13:02</var> GMT</small><br><strong>Update</strong> - Power To Data Hall 4BL is back up. (This is where our Core routers are located)<br />Power To Data Hall 4BH is still down. <br />Power To Data Hall 4BD is still down. (This is where VoIP Services are located)<br /><br />Once we have power back at 4BD, we will look to restore service.</p><p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>12:28</var> GMT</small><br><strong>Update</strong> - We are waiting updates from the Data Centre.</p><p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>11:57</var> GMT</small><br><strong>Identified</strong> - We have lost full power at Volta Data Centre, Onsite engineers are onsite investigating, further update to follow</p><p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>11:44</var> GMT</small><br><strong>Investigating</strong> - Hi. We are currently investigating a issue which may be affecting some customers. <br /><br />Further update will be provided ASAP.</p>tag:status.it-communicationsltd.co.uk,2005:Incident/193370222023-12-05T22:47:43+00:002023-12-05T22:47:45+00:00Carrier Emergency Maintenance (Volta DC)<p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>22:47</var> GMT</small><br><strong>Completed</strong> - Maintenance work now completed.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>20:02</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>17:38</var> GMT</small><br><strong>Scheduled</strong> - From 8pm tonight Carrier Emergency Maintenance will take place.<br /><br />The works involve one of our IP Transit Providers patching fibres into new hardware to ensure we receive a continued stable service.<br /><br />The carrier will also need to repatch fibre for our NNI Leased Lines. <br /><br />Whilst there will be minimal disruption to our network, some leased line customers will lose service until the repatching is completed.</p>tag:status.it-communicationsltd.co.uk,2005:Incident/191886462023-11-24T02:00:06+00:002023-11-24T02:00:06+00:003RD Party Maintenance IP Transit and NNI Services<p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>02:00</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>20:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>21</var>, <var data-var='time'>16:49</var> GMT</small><br><strong>Scheduled</strong> - During this Maintenance, One of our Upstream IP Transit Providers will be relocating their hardware to a new Rack. This will mean we will need to have new Fibres ran to their network. <br /><br />Services Impacted. <br />IP Transit Redundancy - Reduced<br />Some Leased Lines that terminate at Sov House.<br /><br />There will be little to no impact to VoIP Services.</p>tag:status.it-communicationsltd.co.uk,2005:Incident/191885792023-11-22T23:30:54+00:002023-11-22T23:30:55+00:00Sov House - Maintenance<p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>23:30</var> GMT</small><br><strong>Completed</strong> - Maintenance is now completed.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>23:24</var> GMT</small><br><strong>Verifying</strong> - BGP sessions are now starting to establish and traffic passing.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>23:19</var> GMT</small><br><strong>Update</strong> - Router is now back up. 100Gb Downlinks to Core Switches now up. Final Checks before bringing BGP Sessions backup.<br /><br />Customer Leased Lines should now be backup and working.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>22:30</var> GMT</small><br><strong>Update</strong> - We are now in the process of shutting down the Juniper Router so it can be relocated within the rack where there is more space.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>22:27</var> GMT</small><br><strong>Update</strong> - BGP Shutdown on Sov House Router. Where possible traffic re-route to Volta. No impact on VoIP Services. <br /><br />However leased Lines that terminate via Sov House will be without service for up to 40 minutes.<br /><br />Update to follow.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>22:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>21</var>, <var data-var='time'>16:38</var> GMT</small><br><strong>Scheduled</strong> - We will be relocating the Router at Sov House to make space so we can connect the 100Gb MTP patch leads which is the new Router to Switch 100Gb Uplinks/Downlinks. <br /><br />Services based at our Sov-House Data Centre are at risk of losing service during the Maintenance Window. <br /><br />It is estimated the downtime will be up to 30-40 Minutes.<br /><br />Leased Line Customer which terminate at Sov House, will lose Internet Access. Switching hardware will remain up and Data Sov-House to Volta will still be working as normal.</p>tag:status.it-communicationsltd.co.uk,2005:Incident/190696832023-11-16T23:25:52+00:002023-11-16T23:25:53+00:00Network Maintenance Upgrade<p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>23:25</var> GMT</small><br><strong>Completed</strong> - Ok, Phil has had to abort the upgrade at Sov House, the 100Gb LR4 Cables are too big for the rack so will need to plan on moving the Router down a few U's so the cables can fit and rack door remain shut.<br /><br />This will require about 30min of downtime at Sov House to move the router etc. <br /><br />A new maintenance window will be announced when we are ready to relocate the Router.<br /><br />To to Recap. Upgrades at Volta are now fully completed. Sov House needs more time / planning etc of which date to be confirmed.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>22:40</var> GMT</small><br><strong>Update</strong> - Ok, the final upgrade at Volta is now complete. Phil is now heading over to Sov House DC as he cannot see light on the new Fibre's.<br /><br />There is no impact to service at Sov at the moment as the original fibres remain up.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>22:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>11:11</var> GMT</small><br><strong>Scheduled</strong> - This Maintenance window will start at 10pm on 16th November.<br /><br />The Down Links from our Juniper MX204 Routers at Volta and Sov House are both currently running at 20Gbps to our Juniper QFX Core Switches. We have already installed the new Fibres to increase this to 100Gbs to all core switches.<br /><br />The previous maintenance upgraded all switches and site to site links to 100Gbps leaving Uplinks to MX204s remaining at 20Gbps to avoid the routers (with large buffers) microbursting the switches (with small buffers) and causing issues. So the upgrade to full 100Gb had to be done in stages.<br /><br />This Maintenance window is to make live the new fibres and will be the final part of our new 100Gb Backbone Network.<br /><br />This notification is to make you aware that this could cause a disruption to all services while the MX204 Routers reroute and learn the new paths to our core switching hardware.<br /><br />The Maintenance window is set for 2 hours, however the actual disruption is likely to be only a few minutes per switch while the traffic reroutes to the new Fibre.<br /><br />The change of config script will be built in advance so can be applied quicky to reduce the disruption.</p>tag:status.it-communicationsltd.co.uk,2005:Incident/190623842023-11-09T00:30:11+00:002023-11-09T00:30:13+00:00Emergency Maintenance<p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>00:30</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>21:31</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>20:42</var> GMT</small><br><strong>Scheduled</strong> - We will be doing Emergency Maintenance tonight on SW 3 and 4 (2 x 48 x 10Gb Port Dual Stack Switches)<br />Swapping out the Uplink 10Gb SFP Modules to 100Gbps. <br /><br />We will need to reboot the switches as part of this Maintenance but will do one at a time.<br /><br />Therefore the service is at Risk, however due to this being dual stack. Hopefully there should not be any downtime to SIP Services. <br /><br />We will also need to reboot one of the 1GB Dual Stack Switches, this will not have any impact to service.</p>tag:status.it-communicationsltd.co.uk,2005:Incident/190404112023-11-07T01:44:36+00:002023-11-07T01:44:36+00:00Network Upgrade to 100Gb<p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>01:44</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>01:44</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>01:44</var> GMT</small><br><strong>Scheduled</strong> - First part of the upgrade is now completed. Site to Site links are now at 200Gb, Will be more work to be completed</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>00:41</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>00:38</var> GMT</small><br><strong>Scheduled</strong> - We are upgrading our network backbone to 100Gb and Site Links between Volta and Sov House from 20Gb to 200Gb. You may not be able to access some of our services while we are doing the upgrade. Update to follow.</p>tag:status.it-communicationsltd.co.uk,2005:Incident/104789902022-07-06T16:05:59+01:002022-07-06T16:05:59+01:00Urgent Maintenance for Linx Lon1<p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>16:05</var> BST</small><br><strong>Completed</strong> - We can confirm that BGP was restored and have started to see traffic return on on Linx Lon1.</p><p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>15:39</var> BST</small><br><strong>Update</strong> - Fibre now moved over and we are about to bring BGP backup for Linx Lon1.</p><p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>15:27</var> BST</small><br><strong>Update</strong> - BGP is now shutdown.</p><p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>15:07</var> BST</small><br><strong>Update</strong> - We can confirm BGP is about to shutdown.</p><p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>14:16</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>14:14</var> BST</small><br><strong>Scheduled</strong> - We will soon be shutting down Linx Lon1 connection as we urgently need to move the port to a new fibre.<br /><br />We will gracefully shutdown BGP Peering sessions so traffic routes via another network route and once traffic as moved away, we will then shutdown the port and change the fibre over before bringing up the BGP Peering sessions again. This should hopefully not be service affecting due to us shutting down BGP before work is started. <br /><br />The actual downtime of this route should not be for long. however we have set a 4 hour window for when the work is likely to start and complete. if the work is completed early, then we will update to confirm this. We will provide more updates ASAP.</p>tag:status.it-communicationsltd.co.uk,2005:Incident/102596982022-06-12T16:00:00+01:002022-06-13T11:01:05+01:00Network Issues<p><small>Jun <var data-var='date'>12</var>, <var data-var='time'>16:00</var> BST</small><br><strong>Resolved</strong> - Sunday 12th June at around 4pm, our network monitoring system alerted us that some of our servers was not fully accessible over the Internet.<br /><br />We started doing tests and trying to find out what was going on and had to take down parts of the network to see if we can find out the root cause of the issue.<br />We had a team working to early hours of Monday 13th June and not yet been able to find the cause.<br /><br />If anyone has any issues connecting to our network, please contact support so we can investigate.</p>tag:status.it-communicationsltd.co.uk,2005:Incident/84441992021-11-10T13:15:10+00:002021-11-10T13:15:10+00:00London Internet Exchange (Lon1) Loss of Service<p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>13:15</var> GMT</small><br><strong>Resolved</strong> - This is now fixed. It was due to a fibre cable issue outside of our network.</p><p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>12:59</var> GMT</small><br><strong>Investigating</strong> - We have lost access to London Internet Exchange (Lon1). Traffic will automatically reroute until the issue is resolved.<br /><br />We have dispatched an engineer to check our Linx Fibre Cable at the Volta Data Centre before raising this as a fault with the London Internet Exchange (Lon1)</p>tag:status.it-communicationsltd.co.uk,2005:Incident/84048262021-11-05T17:54:19+00:002021-11-05T17:54:20+00:00SoftSwitch PBX CL1 - Server 02 and 03 Issues<p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>17:54</var> GMT</small><br><strong>Resolved</strong> - Server 4 is now added to cluster for WebRTC<br /><br />Full service restored</p><p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>17:37</var> GMT</small><br><strong>Update</strong> - Pbx01 now back in service.</p><p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>17:17</var> GMT</small><br><strong>Update</strong> - PBX-02 is now Operational</p><p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>17:11</var> GMT</small><br><strong>Update</strong> - PBX-03 is now back up and working.</p><p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>17:02</var> GMT</small><br><strong>Update</strong> - We are now in the process of failing the servers over to the good working database so service can be resumed quicker. <br /><br />This will allow us to work on the other databases without continued loss of service.</p><p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>16:59</var> GMT</small><br><strong>Update</strong> - Soft-Switch Cluster 1 - stores its data on 3 seperate databases. Last night we tried to introduce a 4th Database which failed to sync. <br /><br />A fix was applied however casued the other databases replication to crash. The resync is taking its time due to the size of the databases.<br /><br />We continue to work on this until the service is restored. Updates to follow.</p><p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>16:41</var> GMT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>16:40</var> GMT</small><br><strong>Monitoring</strong> - We found a fault with the Databases which stores all the data for Soft-Swtich Cluster 01 and waiting for the databass to catch up with each other before the service will return back to normal.</p>tag:status.it-communicationsltd.co.uk,2005:Incident/79294862021-09-06T14:00:35+01:002021-09-06T14:00:36+01:00Soft-Switch PBX CL2 03 CPU Load Issues<p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>14:00</var> BST</small><br><strong>Resolved</strong> - node is now reporting normal CPU Usage, We will continue to monitor closely.</p><p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>13:27</var> BST</small><br><strong>Monitoring</strong> - Following the changes made to the server node, PBX-CL2-03 is now back in service. Just waiting for the Databases to catch up with any changes while PBX-03 was out of service. <br /><br />We will continue to monitor the node and take action if the CPU load increases again.</p><p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>13:20</var> BST</small><br><strong>Identified</strong> - We are aware of an issue at the moment with one of the server nodes in Soft-Switch Cluster CL2<br /><br />We are investigating at the moment to find out what the cause is.</p>tag:status.it-communicationsltd.co.uk,2005:Incident/76636122021-08-03T02:00:56+01:002021-08-03T02:00:58+01:00Carrier Core Network Upgrades (SN-20210713-3) - VOLTA<p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>02:00</var> BST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>22:02</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>14:08</var> BST</small><br><strong>Scheduled</strong> - As part of our carriers ongoing commitment to stability and performance of their network, they will be performing software updates across their core network.<br /><br />This work will require an outage of around 45 minutes per device.<br /><br />Resilient devices in each site will be upgraded during separate windows to minimise impact across the network.<br /><br />Customers with resilient circuits, or consuming services in other sites should not see any impact to their service. However, all services will be considered at risk for the duration of the window. <br /><br />Customer with non-resilient services will see an impact to their services within the sites included in the provided windows.<br /><br />This should only affect customers with leased lines terminating at Volta, VoIP Services at this location will remain up and usable.</p>tag:status.it-communicationsltd.co.uk,2005:Incident/76219702021-07-29T03:00:57+01:002021-07-29T03:00:57+01:00Service Maintenance 28-07-2021<p><small>Jul <var data-var='date'>29</var>, <var data-var='time'>03:00</var> BST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>22:00</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>08:02</var> BST</small><br><strong>Scheduled</strong> - As part of one of our external carriers ongoing commitment to stability and performance of their network, they will be performing software updates across their core network estate at Sov House DC<br /><br />This work will require an outage of around 45 minutes per device, We are unsure what level of impact this will have. However any leased lines at this location will be affected. <br /><br />VoIP Services should remain up and running during this period due to our network layout etc.<br /><br />Their will be service maintenance conducted between 21:00 UTC and 02:00 UTC on 28/07/2021</p>tag:status.it-communicationsltd.co.uk,2005:Incident/70922802021-05-26T23:32:27+01:002021-05-26T23:32:28+01:00SIP-01.it-communicationsltd.co.uk<p><small>May <var data-var='date'>26</var>, <var data-var='time'>23:32</var> BST</small><br><strong>Completed</strong> - We have now completed the migration.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>22:14</var> BST</small><br><strong>Update</strong> - Database is still coping over at the moment, 300GB of data remaining which will take around 1 hour. <br /><br />No downtime as of yet until the data has finished coping over. Then we have 15min of config to complete and then a cluster reload of which is when the 1 minute of downtime may be noticed. Please ignore any SIP Trunk failed registrations as these will re-register.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>20:01</var> BST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>17:53</var> BST</small><br><strong>Scheduled</strong> - SIP-01.it-communicationsltd.co.uk which currently resides on legacy 3rd party network will be moved to our own network tonight. <br /><br />The IP Address will change but the host name will remain the same.<br /><br />Current IP: 212.38.169.254 (IoMart DC)<br />New IP: 185.131.10.9 (Sov House DC)<br /><br />Customers may see a brief outage of up to 1 minute when we add the server back into the cluster while the system reloads its configuration.</p>tag:status.it-communicationsltd.co.uk,2005:Incident/66556242021-04-01T13:09:44+01:002021-04-01T13:09:44+01:00Call Qualtiy Issues<p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>13:09</var> BST</small><br><strong>Resolved</strong> - We have noticed one of our MPLS Site to Site (Volta to Sov House) Data Centre had suffered 50% Packet Loss. <br /><br />This would have cuased Call Qualtiy Issues and or disconencted calls.<br /><br />We have removed the MPLS Link from our network and will now be taking steps to prevent this in the future.</p>tag:status.it-communicationsltd.co.uk,2005:Incident/65946012021-03-25T23:30:41+00:002021-03-25T23:30:42+00:00London Internet Exchange Lon1 - Outage<p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>23:30</var> GMT</small><br><strong>Resolved</strong> - Linx has now resolved the issues and we have therefore restored our BGP sessions back to the Lon1 Exchange.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>09:25</var> GMT</small><br><strong>Update</strong> - Linx continues to work on the issue and most providers still have their sessions shutdown including us. However this is causing some odd routing issues for many providers. BT network for example are not able to reach some banking websites / networks.<br /><br />So for example, Your Internet Provider may have routed traffic to Barclays Bank via London Linx 1, this is shutdown so traffic routes via BT and then fails. Suspect their maybe wider issues with some networks due to the above.<br /><br />** Response from the London Internet Exchange **<br />Juniper JTAC have made solid progress throughout the last 12 hours, they were able to recreate the corruption issue during the UK night. Though their L2ALD Software Engineering Experts are continuing to investigate the root cause. They will continue their investigations for another few hours before we proceed with our interim working model as detailed in our last update. <br /><br />The network remains stable and we will continue to hold off on any provisioning work for the time being. We have a meeting with Juniper at 13:00 GMT and will update via ops-announce soon after.<br /><br />** Response from the London Internet Exchange **<br /><br />Until London Internet Exchange confims the issue is fullly resolved, most providers including us, will leave sessions shutdown to prevent outages in the event of the London Internet Exchange having issues again.</p><p><small>Mar <var data-var='date'>24</var>, <var data-var='time'>12:17</var> GMT</small><br><strong>Update</strong> - Just a quick update. The London Internet Exchange are stil working on the issue and our BGP Sessions remain shutdown to protect customers from issues.<br /><br />If anyone is unable to reach our network or are having issues with VoIP Services connecting, please let us know.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>17:47</var> GMT</small><br><strong>Identified</strong> - We have had the below statement from the London Internet Exchange.<br /><br />Our BGP Sessions wih Linx's remain shutdown<br /><br />** From London Internet Exchange **<br /><br />LINX LON1 instability issues 23.03.21<br /> <br />At around 11:15 GMT and through to 11:28 GMT, whilst debugging a member issue we saw L2 forwarding disruption on the edge2-thn device impacting a small number of members directly connected.<br /> <br />Furthermore, at around 11:41-11:57 GMT a provisioning change pushed to the edge device edge1-thw appeared to cause a similar L2 forwarding issue. <br /> <br />In both these cases we escalated to the Juniper TAC centre and we decided to halt any further provisioning changes to the LON1 network.<br /> <br />At 15:32 GMT we started to see issues after adding debugging commands as requested by TAC and whilst the L2 issues were quickly mitigated this did cause significant disruption to traffic.<br /> <br />Since the last LON1 disruption at approx. 15:40 GMT the network has remained stable. As a result of adding the debugging we are able to collect more data for the Juniper TAC who are still in the process of investigating this issue. Currently we know a trigger of the issue is any new commits on devices made can affect the L2 forwarding of the device. For this reason we will continue with the suspension of provisioning new services on LON1. <br /> <br />If any members decide to withdraw their peering from LON1 for the time being can you please shutdown your peering and not your ports.<br /> <br />We will provide further updates throughout the evening and to the incident conclusion. <br /><br />** Freom London Internet Exchange **</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>15:49</var> GMT</small><br><strong>Update</strong> - We are now shutting down our session to Linx Lon1 to prevent conenctions from flapping and to ensure traffic is routed away from the exchange.</p><p><small>Mar <var data-var='date'>23</var>, <var data-var='time'>15:47</var> GMT</small><br><strong>Investigating</strong> - London Internet Exchange Lon1 is currently down. This is one of the UKs Main Peering Hub's where most ISPs will connect to for interconenctions between each other.<br /><br />We are currently chasing Linx's for an update. meantile traffic will take anoher path to reach our network.</p>tag:status.it-communicationsltd.co.uk,2005:Incident/63671992021-02-23T17:22:55+00:002021-02-24T13:39:43+00:00Reports of Call Quality Issues<p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>17:22</var> GMT</small><br><strong>Resolved</strong> - We started to receive reports from customers today from around 4pm and resolved by 16:49pm with one-way audio or bad call quality.<br /><br />One of our External Connectivity Providers had an issue within their core network which meant some customers where traffic reaches our network via this particular Internet Route, suffered packet loss.<br /><br />BGP Sessions remained active to the External Connectivity Provider (IP Transit Provider) during the packet loss as packet loss was high enough to cause issues with Voice but not long enough for BGP to pick up on the issue and automatically take down the BGP Session and re-route traffic.<br /><br />We have requested from the affected provider a Reason for Outage Report "RFO" which I have been told will be with us in 24 hours.<br /><br />We will investigate if our monitoring system can be setup to monitor for small packet loss that would not cause the service to completely fail but nevertheless cause issues with Voice Calls and not show up on our monitoring.<br /><br />We will provide a copy of the report once received and an update on what we can do to improve network monitoring.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>16:53</var> GMT</small><br><strong>Update</strong> - We have been informed the external issue is resolved and waiting for a statement to be released so we can have a clear understanding of what caused the issue with the calls. We will provide a reason for the issues shorlty.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>16:35</var> GMT</small><br><strong>Update</strong> - It would now appear the issue is occurring again, we are still investigating.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>16:24</var> GMT</small><br><strong>Monitoring</strong> - It would seem the problem has cleared on its own. We will attempt to find out the cause and provide a further update shortly.<br /><br />Again nothing is showing on our monitoring systems.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>16:10</var> GMT</small><br><strong>Update</strong> - We have confirmed the issue is not on our side of the network. Other phone providers are also having the same issues. We are trying to find out the cause still. Update to follow.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>15:51</var> GMT</small><br><strong>Investigating</strong> - We have received a few reports from customers that they are experiencing poor quality audio or one-way audio. <br /><br />None of our monitoring is showing any issues. <br /><br />We are investigating to see what is causing it.<br /><br />Update to follow.</p>tag:status.it-communicationsltd.co.uk,2005:Incident/62179772021-02-09T22:00:31+00:002021-02-09T22:00:32+00:00Network Upgrade Linx London 2<p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>22:00</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>20:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>10:23</var> GMT</small><br><strong>Update</strong> - Sorry, The work will take place on the 9th Feb not the 8th</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>10:21</var> GMT</small><br><strong>Scheduled</strong> - We will be upgrading our connection for London Linx Exchange at Sov House.<br /><br />The work required will be shifting the Linx's connection from 2 x 1GB Copper Connections to 10Gbps Fibre.<br /><br />This is the last connectivity connection within our network that is still running legacy copper.<br /><br />Once this is migrated our core network will be fully on Fibre.<br /><br />We will shutdown BGP Sessions at 8PM today and wait for traffic to migrate away before doing the required work.</p>tag:status.it-communicationsltd.co.uk,2005:Incident/58051722020-12-19T12:30:31+00:002020-12-19T12:30:33+00:00Colt Capital Markets Service Affecting Maintenance Notification<p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>12:30</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>08:31</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>08:47</var> GMT</small><br><strong>Scheduled</strong> - Colt Capital Markets - LME<br /><br />in order for Colt to maintain the highest levels of availability and service customers, it is sometimes necessary for Colt to carry out planned maintenance on their network. <br /><br />We apologise for any resulting inconvenience and assure you of our efforts to minimise any service disruption. If you require any assistance, please contact Support<br /><br />Planned Work (PW) Ref.: CRQ1-36632315554<br />Start Date & Time: 19/12/2020 08:30:00 GMT<br />End Date & Time: 19/12/2020 12:30:00 GMT<br />Outage Duration: 120 Minutes<br /><br />Justification of the work: This upgrade is part of a planned program, to enhance the availability and performance of Colt’s services. The upgrade is consistent with our vendor’s software release regime and will ensure the appropriate level of ongoing support, as well as provide performance and feature enhancements. The target release has been extensively tested and field-trailed by Colt and the vendor.<br /><br />List of Device: IXLPER02- Interxion London (LON1), Provider Edge Router (Secondary)</p>tag:status.it-communicationsltd.co.uk,2005:Incident/56964732020-12-02T02:00:41+00:002020-12-02T02:00:42+00:00Essential network Upgrade - Service Affecting - 01-12-2020<p><small>Dec <var data-var='date'> 2</var>, <var data-var='time'>02:00</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>20:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>12:35</var> GMT</small><br><strong>Scheduled</strong> - We will be removing the Brocade Switches and Router at the Sov House Data Location and replacing with Juniper.<br /><br />We will also be upgrading one of our IP Transit (Bandwidth Providers) from 1Gbps to 10Gbps<br /><br />While the upgrade takes place there is a risk to service.</p>tag:status.it-communicationsltd.co.uk,2005:Incident/55245672020-11-13T00:00:33+00:002020-11-13T00:00:34+00:00Essential network upgrade - Service Affecting<p><small>Nov <var data-var='date'>13</var>, <var data-var='time'>00:00</var> GMT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>12</var>, <var data-var='time'>20:00</var> GMT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>12</var>, <var data-var='time'>17:19</var> GMT</small><br><strong>Update</strong> - Due to a delay with some of the hardware. the full migration will now not take place tonight. Instead we are going to use this maintenance window to add additional redundancy for our Leased Line Customers by adding in an additional NNI Port.<br /><br />We will open a new maintenance window ASAP to complete the full migration off of Brocade for Sov House DC.</p><p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>15:31</var> GMT</small><br><strong>Scheduled</strong> - We will be removing the Brocade Switches and Router at the Sov House Data Location and replacing with Juniper.<br /><br />While we have a 4 hour window open, we only expect the actual work to take half hour due to everything being setup in advance.<br /><br />We will also be upgrading one of our IP Transit (Bandwidth Providers) from 1Gbps to 10Gbps<br /><br />While the upgraded takes place there is a risk to service. However any downtime should be limited by the time it takes to swap the network cables to the new hardware so all being well should be just a few seconds per device / server.</p>tag:status.it-communicationsltd.co.uk,2005:Incident/53135852020-10-13T22:29:25+01:002020-10-13T22:29:26+01:00Incident at the Volta Data Centre<p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>22:29</var> BST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>11:40</var> BST</small><br><strong>Monitoring</strong> - The issue seems to have cleared now, however, service is still at risk until we are told officially the problem is resolved.<br /><br />A full RFO will be released.</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>10:13</var> BST</small><br><strong>Update</strong> - We are starting to see the packet loss issue improve now and the faults showing on our Internal monitoring are starting to clear.</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>10:09</var> BST</small><br><strong>Update</strong> - Our IP Transit Provider continues to work on the fault that is causing issues for our customers. We are chasing for updates</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>09:49</var> BST</small><br><strong>Update</strong> - It would appear to be related to one of our IP Transit Providers at Volta Data Centre having issues. This is causing packet loss into and out of our network. The IP Transit Provider is aware of the issue and is currently working on a fix. Update to follow.</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>09:39</var> BST</small><br><strong>Update</strong> - Our network team are still investigating to find the cause of the fault which may be causing VoIP Issues.</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>09:25</var> BST</small><br><strong>Investigating</strong> - We are investigating an issue that appears to be at the Volta Data Centre. Update to follow.</p>