Outage : Problem with internet services – DDoS attacks on 28/08-31/08/20 : SOLVED

Dear customers,

We have experienced an outage with internet services (traffic drops) due to several DDoS attacks*** on our network. The attacks occurred at different hours and were causing intermittent connection, speed issues and packet loss. The heaviest one took place on Monday 31/08 and lasted from 11h15 until 18h45. Outside of those hours, everything was normal. Our Network Operations team has found and applied (on 01/09 at 1h29) a permanent solution to prevent and immediately mitigate such attacks in the future.

Please find all details below:

Start*: 2020-08-28 18:00*
End*: 2020-09-01 01:29*
Duration: 1) 2020-08-28 18:00 – 2020-08-28 19:00

2) 2020-08-30 20:30 – 2020-08-30 20:45

3) 2020-08-30 22:20 – 2020-08-31 00:55

4) 2020-08-31 04:05 – 2020-08-31 05:05

5) 2020-08-31 11:15 – 2020-08-31 18:45

Impacted services: all internet services
Affected Areas**: all areas

EDIT 12:30*: We have been under attack again since approximately 11:15. Access to our issues blog was limited so we were unable to keep you updated. Our apologies. We are doing our best to mitigate and solve the problem.

EDIT 2: 15h12*: Seems that the last attack is still ongoing.  We are working hard to try to minimize the impact. Rebooting your modem/router to receive new DNS servers can help (but not guaranteed).  In general surfing should be possible, only certain destinations will be difficult to reach.  We keep you posted.

EDIT 3: 20h25*: The last attack finished at 18:45. We did our best to mitigate what was possible but the overload was extensive. Edpnet DNS servers are currently available again. We are working very hard with an external party towards providing a permanent solution to minimize the impact of any further attacks.

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time
** Check the different areas & zones on our support site

*** DDoS (Distributed-Denial-of-Service- attack) is kind of a cyber-attack, when malicous users seek to make a machine or network resource unavailable by flooding the provider or target with superfluous requests, which results in systems overload and connectivity issues


Outage : Problem with internet services – SOLVED

Dear customers,

We are currently experiencing an outage with internet services (traffic drops). Our Network Operations team is investigating the outage.

Further details on the affected areas and an estimated duration will be released as soon as possible.

EDIT 19:40*: Our DNS-servers were the target of a concerted DDOS-attack for approximately one hour (18:00 – 19:00), involving large amounts of traffic and major network saturation. At this time the attack has cleared up and we are investigating further.

EDIT 30/8 20:35*: It seems like we’re being attacked again since a few minutes. Our Network Operations team is investigating. More updates will follow soon.

EDIT 20:45*: The attack was mitigated after some actions from our Network Operations team. It was smaller than last time, but still very much able to knock out our DNS-servers. We’re working to restore all services.

EDIT: 21:15*: All services are restored.

EDIT: 31/8 5h05 We experienced other waves of DDoS attacks between 30/8 22h20 and 31/08 05h05. For the moment, the attacks are fully mitigated.

We apologize for the inconveniences.

Best regards
The edpnet team


Info: Fake Non-delivery notifications

Dear customers,

We would like to inform you about a new phishing wave that is aimed at both customers and non-customers.

Customers receive emails pretending to come from a mailserver / Microsoft Online Service, etc. The message tells you that your incoming mails have been placed on hold until you click on a link.

Here is an example:

This is a blackmail spam and opening the link may cause a leak of your logins.

Please do not reply to those emails and do not open any files or click on any links. For more information, please, consider reading through our support article on this subject

https://www.edpnet.be/en/support/troubleshooting/internet/learn-about-security/phishing.html

If you have any questions left, please do not hesitate to contact us.

Best regards
The edpnet team


Info: Phishing pop-up with edpnet logo

Dear customers,

We would like to inform you about a new phishing wave – customers see a pop-up with edpnet logo shown up while surfing the web. The message claims that edpnet is running a promotion on the occasion of its anniversary (or a satisfaction survey, contest, etc.) and is offering some gifts to the customers.

This is a fake phishing pop-up and is not coming from edpnet.

Please do not click on any link and do not reply to that message!

For more information, please, consider reading through our support article on this subject

https://www.edpnet.be/en/support/troubleshooting/internet/learn-about-security/how-do-i-block-phishing-pop-ups.html

If you have any questions left, please do not hesitate to contact us.

Best regards
The edpnet team


Maintenance planned on 10/12/2018 – impacted services: internet connectivity and telephony – region : Ghent : Closed

Dear customers,

Proximus has planned a maintenance which may affect your edpnet services.

Please find all details below:

Start: 2018-12-10 01:00*
End: 2018-12-10 06:00*
Duration: potential impact : up to 30 minutes
Impacted services: internet connectivity and telephony
Affected Areas: Mariakerke, Wondelgem, Drongen, Vinderhoute, Vosselare, Nevele, Merendree

 

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time


Outage: Telephony: Solved

Dear customers,

We had experienced an outage with Telephony services. There was some delay in connecting calls, some calls failed. Our Network Operations team has solved the issue.

Please find all details below:

Start: 2018-06-04 09:00*
End: 2018-06-04 11:10*
Duration: 2h10
Impacted services: Telephony
Affected Areas: all

Update: 2018-06-04 10:25 Repair Database ongoing

Update: 2018-06-04 09:41 Core switch reboot done

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time


Outage: Outgoing telephony (Solved)

Dear customers,

We are currently experiencing an outage with outbound telephony.  Our Network Operations team is investigating this outage.  Further updates will be released as soon as possible.

Please find all details below:

Start: 2017-06-13 11:55*
End: 2017-06-13 14:25*
Duration: 2h30m
Impacted services: Outgoing telephony
Affected Areas: All areas

Update 13:30: Our NOC-team has found the cause of the issue and is currently working to resolve it. We are working to minimize the impact. Edpnet telephony (VOIP) customers and virtual PBX customers are not impacted. We are aiming for an expected time of resolution of 1 hour from now.

Update 14:25: The telephony database has been rebuilt and telephony has been successfully tested with some impacted customers. Please contact us if you are still experiencing any issues and have already rebooted your hardware.

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time


Info: Overview of the latest outages and solutions

Dear customers and partners,

As you all know, edpnet has gone through a number of serious outages lately. The initial power outage exerted a huge impact on our core infrastructure which caused, unfortunately, much tension to our customers and ourselves. We understand that it is hard to justify the confidence in such circumstances and want to apologize for all inconvenience. Strong measures have already been taken to reduce a risk of further outages and guarantee the stability of our services.

As open communication is very important for us, we would like to provide you with a detailed overview of what exactly happened and what we have done/are going to do to prevent these situations in future and improve our service. Please, find the overview below.

Monday 26/09/2016 4:49: the data center of edpnet Sint-Niklaas lost power, UPS system and power generator failed.

Monday 26/09/2016 10:19: the power was restored, systems were back online. Due to the many issues to get the power online, we provided a backup power feed from our office power feed to the most critical systems

Monday 26/09/2016 11:00: mail load balancers were dead, probably due to intermediate power cuts. These devices were replaced.

Monday 26/09/2016 20:00: the power was lost again in the data center, critical applications such as voice services and main router were still running. The interruption caused an issue on incoming calls.

Monday 26/09/2016 20:15: The root cause of the power issue was found, the UPS itself caused the power cuts. We bypassed the UPS and restored power.

Monday 26/09/2016 21:30: incoming calls were running fine again.

After the power cut, we noticed some strange behavior on our Voice switch 2, several maintenances were performed to have the system stable again, no more issues were noticed after these maintenances.

Sunday 2/10/2016 23:30: a DDoS attack starts which causes connectivity problems.

Monday 3/10/2016 9:30: The DDoS target is found and disabled. Connectivity restored.

Tuesday 4/10/2016 0:00: Planned maintenance to replace the broken UPS, and installing an external bypass system so future UPS maintenance shouldn’t give an interruption.

Wednesday 5/10/2016 7:44: A database issue occurred on Voice switch 2.

Wednesday 5/10/2016 8:06: The database is restored, voice is operational.

Thursday 6/10/2016 1:46: Voice switch 2 crashes, is completely unavailable. Voice switch 1 isn’t taking over causing voice calls not to arrive, and outgoing calls not to be made.

Thursday 6/10/2016 6:43: manually switched all voice traffic towards Voice switch 1, incoming and outgoing calls seem to work fine.

Thursday 6/10/2016 9:00: complaints arrive that no incoming calls are possible again, it seems Voice Switch 1 only receives 50% of the incoming calls

Thursday 6/10/2016 10:57: restored Voice switch 2 on spare hardware, all calls are arriving no more voice issues.

Thursday 6/10/2016 11:30: Proximus and supplier found the issue with the failing redundancy of the voice switches, a new maintenance will be planned soon to make the necessary improvements.

List of improvements to be made:

  1. UPS replacement (done)
  2. External bypass system to avoid outage due to UPS works (done)
  3. Move certain critical services to our data center in Interxion Brussels (ongoing)
  4. Install a permanent B-power feed in our data center in Sint-Niklaas (ongoing)
  5. Adapt voice service for correct redundancy (done)

 

If you have any questions or remarks left, please, do not hesitate to contact us.

Best regards
The edpnet team


Outage: Service degradation / High latency

Dear customers,

We are currently experiencing an outage with internet services, due to paths cut between Brussels and Amsterdam. Traffic is rerouted but customers may notice a service degradation and high latency. The outage is currently under investigation by our NOC team. Further updates will be released as soon as possible.

Please find all details below:

Start: 2016-08-11 11:46 AM
End: n/a
Duration: n/a
Impacted services: all internet services: possible quality degradation, high latency
Affected Areas: all zones

 

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time

 

Update 22:45 – Issue solved. Traffic is taking the normal path again.


Outage: all data services (internet, telephony, …)

Dear customers,

We have experienced an outage with DSL services impacted by the maintenance of last night. For most customers, the problem was solved when they restarted their modems. The outage has been fixed by our NOC team.

Please find all details below:

Start: 2016-06-07 2:09 AM*
End: 2016-06-07 9:47 AM*
Duration: 7,5 hours
Impacted services: All data services
Affected Areas: All DSL customers in Belgium

 

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time