{"id":2971,"date":"2020-05-19T14:52:53","date_gmt":"2020-05-19T12:52:53","guid":{"rendered":"http:\/\/issues.edpnet.be\/?p=2971"},"modified":"2020-05-19T15:20:38","modified_gmt":"2020-05-19T13:20:38","slug":"info-report-on-the-outage-of-13-05-2020","status":"publish","type":"post","link":"https:\/\/issues.edpnet.be\/?p=2971","title":{"rendered":"Info: Report on the outage of 13\/05\/2020"},"content":{"rendered":"<p>Dear customers,<\/p>\n<p>Since we strive to maintain an open communication with our customers, we owe you a short report about our outage of last week.<\/p>\n<p>On Tuesday 12\/05\/2020 at the end of the day we saw an issue with a card in our POP in Germany on our European fiber ring. Luckily all traffic is rerouted via the other side of the ring. One of our network engineers leaves the next day, Wednesday 13\/05\/2020, in the early morning to Germany to replace the defective card.<\/p>\n<p>Between Tuesday night and Wednesday morning, 13\/05\/2020 Eurofiber had a planned fiber maintenance in Antwerp. To be clear, our fibers were not involved in this maintenance.\u00a0 But at 5h30 an Eurofiber engineer accidentally jams one of our fiber cables between a cabinet door.\u00a0 This caused a second cut\/outage on our fiber ring.\u00a0 This double cut\/outage resulted in a disconnection from our Belgian network and our locations in Amsterdam and Germany. As a big part of our traffic is transmitted to Amsterdam and Frankfurt, we didn\u2019t have enough capacity anymore in Belgium to handle peak traffic.<\/p>\n<p>At 9h06, the Eurofiber engineers located the issue with our jammed fiber and solved it. At that moment we were back online but issues occurred when we started to rebuild our (fiber)\u00a0<a href=\"https:\/\/en.wikipedia.org\/wiki\/Wavelength-division_multiplexing\">wavelengths<\/a>\u00a0towards Amsterdam and Germany. As traffic was growing, our internal links inside the Belgian network were oversaturated.\u00a0 This caused time-outs\u00a0to\u00a0websites for many of our DSL customers.<\/p>\n<p>Our network engineer arrived at 12h20\u00a0in our\u00a0German site.\u00a0 The card was replaced within half an hour.\u00a0 At 13h01 we were able to get the wavelengths up and running as it should and our European ring was again fully redundant.\u00a0 Almost 1 hour later, at 14h15 our\u00a0<a href=\"https:\/\/en.wikipedia.org\/wiki\/Border_Gateway_Protocol\">BGP<\/a>\u00a0communications with the outside world went back to normal after being disturbed due to the high amounts of attempts to get the fiber wavelengths back up.<\/p>\n<p><strong>What will we do to prevent this in the future?<\/strong><\/p>\n<p>We will install a backup for our\u00a0<a href=\"https:\/\/en.wikipedia.org\/wiki\/Out-of-band_management\">out of band management<\/a>\u00a0interface which manages our equipment for traffic transportation, so even when there are 2 cuts on the ring, we can reach them from outside our own networks.\u00a0This would result in a shorter repair time and better overview in case of issues like this.<\/p>\n<p>Once again, we apologize for any inconveniences caused.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Dear customers, Since we strive to maintain an open communication with our customers, we owe you a short report about our outage of last week. On Tuesday 12\/05\/2020 at the end of the day we saw an issue with a card in our POP in Germany on our European fiber ring. Luckily all traffic is [&hellip;]<\/p>\n","protected":false},"author":15,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[27,28],"tags":[72,60,71],"class_list":["post-2971","post","type-post","status-publish","format-standard","hentry","category-belgium","category-info","tag-belgium","tag-info","tag-internet"],"_links":{"self":[{"href":"https:\/\/issues.edpnet.be\/index.php?rest_route=\/wp\/v2\/posts\/2971","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/issues.edpnet.be\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/issues.edpnet.be\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/issues.edpnet.be\/index.php?rest_route=\/wp\/v2\/users\/15"}],"replies":[{"embeddable":true,"href":"https:\/\/issues.edpnet.be\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2971"}],"version-history":[{"count":4,"href":"https:\/\/issues.edpnet.be\/index.php?rest_route=\/wp\/v2\/posts\/2971\/revisions"}],"predecessor-version":[{"id":2981,"href":"https:\/\/issues.edpnet.be\/index.php?rest_route=\/wp\/v2\/posts\/2971\/revisions\/2981"}],"wp:attachment":[{"href":"https:\/\/issues.edpnet.be\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2971"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/issues.edpnet.be\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2971"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/issues.edpnet.be\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2971"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}