Outage: Power outage datacenter Sint-Niklaas – No impact : Closed

Issue closed.

Dear customers,

We are currently experiencing an outage at the datacenter in Sint-Niklaas.  Power generator A-feed and B-feed successfully took the load.  The power company is fixing the outage as we speak. Further updates will be released as soon as possible.

No impact for customers.

Please find all details below:

Start: 2017-06-27 16:24*
End: 2017-06-27 16:59*
Duration: 35min
Impacted services: None
Affected Areas: Colocation Sint-Niklaas

 

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time


Outage: Outgoing telephony (Solved)

Dear customers,

We are currently experiencing an outage with outbound telephony.  Our Network Operations team is investigating this outage.  Further updates will be released as soon as possible.

Please find all details below:

Start: 2017-06-13 11:55*
End: 2017-06-13 14:25*
Duration: 2h30m
Impacted services: Outgoing telephony
Affected Areas: All areas

Update 13:30: Our NOC-team has found the cause of the issue and is currently working to resolve it. We are working to minimize the impact. Edpnet telephony (VOIP) customers and virtual PBX customers are not impacted. We are aiming for an expected time of resolution of 1 hour from now.

Update 14:25: The telephony database has been rebuilt and telephony has been successfully tested with some impacted customers. Please contact us if you are still experiencing any issues and have already rebooted your hardware.

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time


Outage: Internet services Brussels region (solved)

Dear customers,

We are currently experiencing an outage with voice and internet region Brussels.  Our Network Operations team is investigating this outage.  Further updates will be released as soon as possible.

Please find all details below:

Start: 2017-05-09 20:03*
End: 2017-05-09 21:40*
Duration: 1h 37
Impacted services: voice and internet region Brussels
Affected Areas: Service area Brussels: 02

edit 20:46: Proximus confirms power outage in colocation service area Brussels, both A and B feed down, technicians onsite to resolve the issue.

edit 21:48: Power feeds are restored, services back to normal.

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time


Outage: Internet services (solved)

Dear customers,

We are currently experiencing an outage with Internet services. The outage is currently under investigation by our NOC team. Further updates will be released as soon as possible.

Please find all details below:

Start: 2017-04-25 11:30*
End: 2017-04-25 12:10*
Duration: 40 minutes
Impacted services: All data services (Internet, telephony)
Affected Areas: Brussels, Antwerpen, Limburg, Flemish Brabant

Update 25/04 12:10: All lines are restored. Reason of outage: unexpected issue on the server.

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time


Outage: DNS and mail services

Dear customers,

We have experienced an outage with one of our servers, causing issues with DNS and mail delivery to be stalled for a while. All impacted services have been restored by our NOC team at around 7 PM, our mail relays cleared out their backlogs.

Please find all details below:

Start: 2016-02-19 14:00*
End: 2016-02-19 19:00*
Duration: 5 hours
Impacted services: DNS and mail services
Affected Areas: All areas

 

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time


Update: Outage Closed: KPN link for fiber is down

Update: Outage cleared

Dear customers,

We are currently experiencing an outage with our KPN link for fiber. The outage is currently under investigation by our NOC team. Further updates will be released as soon as possible.

Please find all details below:

Start: 2017-01-26 10:38
End: 2017-01-26 10:56
Duration: 18 minutes
Impacted services: NL Fiber
Affected Areas: Amsterdam

 

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time


Outage: NL internet services: Solved

Update: Our services will be back online within 30 minutes

Dear customers,

We are currently experiencing an outage with internet services in the Netherlands. The outage is currently under investigation by our NOC team. Further updates will be released as soon as possible.

Please find all details below:

Start: 2016-12-29 15:30
End: 2016-12-29 16:20
Duration: 50 min
Impacted services: Internet services
Affected Areas: NL

 

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time


Outage: All telephony services (solved)

Dear customers,

We are currently experiencing an outage with telephony. The outage is currently under investigation by our NOC team. Further updates will be released as soon as possible.

Please find all details below:

Start: 2016-11-18 13:32*
End: 2016-11-18 20:00*
Duration: 6:28
Impacted services: All telephony services
Affected Areas: All areas

Update 18/11 20:00: Full capacity is restored, multiple link resets were needed to restore all links. The problem will be investigated and corrective measures will be taken to avoid this issue in the future.

Update 18/11 14:56: The hardware replacement did not have the required effect and telephony is currently running at reduced capacity. We are still looking into solving the issue as soon as possible.

Update 18/11 13:52: The fault was found (hardware failure) and the piece has been replaced. We are following up and monitoring the issue.

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time


Outage: Internet services (solved)

Dear customers,

We have experienced an outage with internet services in the Wavre / Charleroi areas. The outage is currently under investigation by our NOC team. Further updates will be released as soon as possible.

Please find all details below:

Start: 2016-11-3 16:15*
End: 2016-11-3 16:40*
Duration: 25 minutes
Impacted services: All services
Affected Areas: 071GIL: 010, 060, 064, 065, 067, 068, 069, 071

Update 4/11/2016 10:15: the solution has been found, proximus is resetting the communication paths one by one, lines should be starting to get back online one by one.  We advise you to reboot your router if you do not see any changes in the status of your connection after 30 min.

Update 4/11/2016 8:08: some lines remain offline, rebooting doesn’t seem to help. These cases are under investigation, we do our utmost to have the issue resolved.

Update 3/11/2016 21:45: Root cause found, works were performed to extend our capacity, and a configuration mistake was made by proximus. The configuration is adapted and we recommend rebooting your device to regain internet access.

Update 3/11/2016 17:00 Recovery of offline customers is our top priority. A lot of the affected lines are back up, but there are still some down.

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time


Info: Overview of the latest outages and solutions

Dear customers and partners,

As you all know, edpnet has gone through a number of serious outages lately. The initial power outage exerted a huge impact on our core infrastructure which caused, unfortunately, much tension to our customers and ourselves. We understand that it is hard to justify the confidence in such circumstances and want to apologize for all inconvenience. Strong measures have already been taken to reduce a risk of further outages and guarantee the stability of our services.

As open communication is very important for us, we would like to provide you with a detailed overview of what exactly happened and what we have done/are going to do to prevent these situations in future and improve our service. Please, find the overview below.

Monday 26/09/2016 4:49: the data center of edpnet Sint-Niklaas lost power, UPS system and power generator failed.

Monday 26/09/2016 10:19: the power was restored, systems were back online. Due to the many issues to get the power online, we provided a backup power feed from our office power feed to the most critical systems

Monday 26/09/2016 11:00: mail load balancers were dead, probably due to intermediate power cuts. These devices were replaced.

Monday 26/09/2016 20:00: the power was lost again in the data center, critical applications such as voice services and main router were still running. The interruption caused an issue on incoming calls.

Monday 26/09/2016 20:15: The root cause of the power issue was found, the UPS itself caused the power cuts. We bypassed the UPS and restored power.

Monday 26/09/2016 21:30: incoming calls were running fine again.

After the power cut, we noticed some strange behavior on our Voice switch 2, several maintenances were performed to have the system stable again, no more issues were noticed after these maintenances.

Sunday 2/10/2016 23:30: a DDoS attack starts which causes connectivity problems.

Monday 3/10/2016 9:30: The DDoS target is found and disabled. Connectivity restored.

Tuesday 4/10/2016 0:00: Planned maintenance to replace the broken UPS, and installing an external bypass system so future UPS maintenance shouldn’t give an interruption.

Wednesday 5/10/2016 7:44: A database issue occurred on Voice switch 2.

Wednesday 5/10/2016 8:06: The database is restored, voice is operational.

Thursday 6/10/2016 1:46: Voice switch 2 crashes, is completely unavailable. Voice switch 1 isn’t taking over causing voice calls not to arrive, and outgoing calls not to be made.

Thursday 6/10/2016 6:43: manually switched all voice traffic towards Voice switch 1, incoming and outgoing calls seem to work fine.

Thursday 6/10/2016 9:00: complaints arrive that no incoming calls are possible again, it seems Voice Switch 1 only receives 50% of the incoming calls

Thursday 6/10/2016 10:57: restored Voice switch 2 on spare hardware, all calls are arriving no more voice issues.

Thursday 6/10/2016 11:30: Proximus and supplier found the issue with the failing redundancy of the voice switches, a new maintenance will be planned soon to make the necessary improvements.

List of improvements to be made:

  1. UPS replacement (done)
  2. External bypass system to avoid outage due to UPS works (done)
  3. Move certain critical services to our data center in Interxion Brussels (ongoing)
  4. Install a permanent B-power feed in our data center in Sint-Niklaas (ongoing)
  5. Adapt voice service for correct redundancy (done)

 

If you have any questions or remarks left, please, do not hesitate to contact us.

Best regards
The edpnet team