Maintenance planned on 20/12/2016 – impacted services: 5 min loss of connectivity (closed)

Update 20/12/2016 – 06:18

Dear customers,

This  maintenance was successfully completed.

Best regards
The edpnet team

Dear customers,

Edpnet has planned a maintenance in order to migrate the lines to a new port.

Customers will need to reconnect. If some customers are unable to connect automatically, they may need to reboot their modem.

Please find all details below:

Start: 2016-12-20 05:00*
End: 2016-12-20 06:00*
Duration: 5 minutes
Impacted services: All data services (internet, telephony, …)
Affected Areas: All areas

 

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time


Maintenance planned on 07/10 – impacted services: All telephony services (closed)

Update 07/10/2016 – 02:48:

Dear customers,

Last night’s maintenance was successfully completed. Redundancy tests were performed by bringing down each voice switch.

One switch failed during these tests and corrective measures were taken. After these measures all tests were conducted again and finished 100% successfully.

Best regards
The edpnet team

 

Initial post:

Dear customers,

Edpnet has planned a maintenance in order to provide full redundancy on our telephony services in the future.

This maintenance may affect your services.

Please find all details below:

Start: 2016-10-07 01:00*
End: 2016-10-07 03:00*
Duration: Approx. 60 – 90 minutes
Impacted services: All telephony services
Affected Areas: All

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time


Info: Overview of the latest outages and solutions

Dear customers and partners,

As you all know, edpnet has gone through a number of serious outages lately. The initial power outage exerted a huge impact on our core infrastructure which caused, unfortunately, much tension to our customers and ourselves. We understand that it is hard to justify the confidence in such circumstances and want to apologize for all inconvenience. Strong measures have already been taken to reduce a risk of further outages and guarantee the stability of our services.

As open communication is very important for us, we would like to provide you with a detailed overview of what exactly happened and what we have done/are going to do to prevent these situations in future and improve our service. Please, find the overview below.

Monday 26/09/2016 4:49: the data center of edpnet Sint-Niklaas lost power, UPS system and power generator failed.

Monday 26/09/2016 10:19: the power was restored, systems were back online. Due to the many issues to get the power online, we provided a backup power feed from our office power feed to the most critical systems

Monday 26/09/2016 11:00: mail load balancers were dead, probably due to intermediate power cuts. These devices were replaced.

Monday 26/09/2016 20:00: the power was lost again in the data center, critical applications such as voice services and main router were still running. The interruption caused an issue on incoming calls.

Monday 26/09/2016 20:15: The root cause of the power issue was found, the UPS itself caused the power cuts. We bypassed the UPS and restored power.

Monday 26/09/2016 21:30: incoming calls were running fine again.

After the power cut, we noticed some strange behavior on our Voice switch 2, several maintenances were performed to have the system stable again, no more issues were noticed after these maintenances.

Sunday 2/10/2016 23:30: a DDoS attack starts which causes connectivity problems.

Monday 3/10/2016 9:30: The DDoS target is found and disabled. Connectivity restored.

Tuesday 4/10/2016 0:00: Planned maintenance to replace the broken UPS, and installing an external bypass system so future UPS maintenance shouldn’t give an interruption.

Wednesday 5/10/2016 7:44: A database issue occurred on Voice switch 2.

Wednesday 5/10/2016 8:06: The database is restored, voice is operational.

Thursday 6/10/2016 1:46: Voice switch 2 crashes, is completely unavailable. Voice switch 1 isn’t taking over causing voice calls not to arrive, and outgoing calls not to be made.

Thursday 6/10/2016 6:43: manually switched all voice traffic towards Voice switch 1, incoming and outgoing calls seem to work fine.

Thursday 6/10/2016 9:00: complaints arrive that no incoming calls are possible again, it seems Voice Switch 1 only receives 50% of the incoming calls

Thursday 6/10/2016 10:57: restored Voice switch 2 on spare hardware, all calls are arriving no more voice issues.

Thursday 6/10/2016 11:30: Proximus and supplier found the issue with the failing redundancy of the voice switches, a new maintenance will be planned soon to make the necessary improvements.

List of improvements to be made:

  1. UPS replacement (done)
  2. External bypass system to avoid outage due to UPS works (done)
  3. Move certain critical services to our data center in Interxion Brussels (ongoing)
  4. Install a permanent B-power feed in our data center in Sint-Niklaas (ongoing)
  5. Adapt voice service for correct redundancy (done)

 

If you have any questions or remarks left, please, do not hesitate to contact us.

Best regards
The edpnet team


Outage: Service degradation / High latency

Dear customers,

We are currently experiencing an outage with internet services, due to paths cut between Brussels and Amsterdam. Traffic is rerouted but customers may notice a service degradation and high latency. The outage is currently under investigation by our NOC team. Further updates will be released as soon as possible.

Please find all details below:

Start: 2016-08-11 11:46 AM
End: n/a
Duration: n/a
Impacted services: all internet services: possible quality degradation, high latency
Affected Areas: all zones

 

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time

 

Update 22:45 – Issue solved. Traffic is taking the normal path again.


Outage: telephony

Dear customers,

We are currently experiencing an outage with telephony services. The outage is currently under investigation by our NOC team. Further updates will be released as soon as possible.

Please find all details below:

Start: 2016-08-08 02:30 PM
End: 2016-08-08 04:00 PM
Duration: 1 h 30 min
Impacted services: telephony
Affected Areas: all

Update 15h40 : Currently almost all calls are working. Permanent fix expected within a few minutes

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time


Outage: Telephony services working unstable

Update 24/05/2016 11:30 AM – The issue has been solved

Dear customers,

We are currently experiencing an outage with one of our telephony servers.

Customers may notice troubles with their telephony services (VoIP, SIP-trunking).

The outage is currently under investigation by our NOC team. Further updates will be released as soon as possible.

Please find all details below:

Start: 2016-05-24 11:10 AM*
End: 2016-05-24 11:30 AM
Duration: 20 minutes
Impacted services: Telephony services (VoIP, SIP-trunking)
Affected Areas: All customers in Belgium with telephony services at edpnet

 

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time


Maintenance planned on 07/06/2016 – impacted services: all DSL services in Belgium

Dear customers,

Edpnet has planned a maintenance ont its remote servers.

DSL customers will face few interruptions during the maintenance.

Please find all details below:

Start: 2016-06-07 02:00 AM*
End: 2016-06-07 06:00 AM*
Duration: Up to 15 minutes
Impacted services: All data services (internet, telephony, …)
Affected Areas: All DSL customers in Belgium. In case of disconnection, we advise to restart hardware (modem/router)

 

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time


Maintenance planned on 31/05/2016 – impacted services: all DSL services in Netherlands and possibly in Belgium

Dear customers,

Edpnet has planned a maintenance on its remote servers.

We’ll be keeping the downtime to a minimum, but expect several reconnections during the maintenance.

Please find all details below:

Start: 2016-05-31 03:00 AM*
End: 2016-05-31 06:00 AM*
Duration: 15 minutes
Impacted services: All data services (internet, telephony, …)
Affected Areas: Customers connections in Netherlands will be dropped for 15 minutes. Possible loss of connection for users in Belgium. In case of disconnection, we advise to restart hardware (modem/router).

 

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time


Outage: Walloon Brabant, Hainaut – all services impacted

Dear customers,

Edpnet was experiencing a short outage with all DSL services in Walloon Brabant, Hainaut due to a power issue on the network.

Please find all details below:

Start: 2016-05-17 14:00 PM*
End: 2016-05-17 14:30 PM*
Duration: 30 minutes
Impacted services: All DSL services: Internet, VOIP telephony
Affected Areas:
Walloon Brabant, Hainaut: 010, 060, 064, 065, 067, 068, 069, 071

 

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time


Outage: Namur area – all services impacted

Update 02/05/2016 – 10h01: Impacted customers went back online on 29/04 by 12h40. The full restoration of Proximus network has been finished on Saturday noon, all the services are functioning flawlessly since then.

Dear customers,

We are currently experiencing an outage with all internet services in the Namur area. A fire caused damage to the Proximus infrastructure. Further updates will be released as soon as possible.

Please find all details below:

Start: 2016-04-29 09:25 AM*
End: 2016-04-29 12:40 PM*
Duration: 3 hours 15 minutes
Impacted services: All internet services
Affected Areas: Namur area

 

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time