8 January 2014 – Mail Exchange Issue

Maintenance to improve stability.

Update 13 – 21/01 17:03
We schedule a maintenance, tomorrow morning at 05:00, to improve the performance and stability of our mail servers. During the maintenance all mail service will be temporary unavailable. No mail will be lost.

Maintenance will start on 22/01 around 05:00 and will take about 30 minutes.

Update 12 – 20/01 10:30
After the works of this weekend it seems that we have found the origin of the slow responsiveness of the mail servers. By replacing the defect disk controller in our main storage server the speed of our mail servers increased drastically. Monday morning we had our first real stress test by you as customer (Monday morning is the most busy time frame in the week for our mail servers) and the response time of our mail service stayed normal. We keep on monitoring our mail system but we presume this issue finally is solved.

Update 11 – 18/01 10:29
Yesterday evening we found another problem with the storage systems, we completely replaced one of the storage systems, this device is now replicating  data, and should be finished today. We also needed to replace a defective disk controller in another storage system, this night this controller started working as it should again. Due to the fact that the load is lower in the weekend and at night, everything should anyway work optimal. We need to wait until Monday to see if the problem rises again.  We are very sorry for the lack of communication yesterday, we were working very hard, but had no updates.

Update 10 – 17/01 10:57

The overnight monitoring didn’t give the expected result. Therefore our technical team will perform another maintenance at 11h. We keep you up to date as soon as we have more feedback. For your information: meanwhile webmail is back online. Please clear your browser cache if you still arrive on the issues page when connecting to webmail.

Update 9 – 16/01 16:40
Together with our software supplier we are currently implementing a new possible solution. During the next hours and night we will monitor this new implementation to see if it gives the necessary speed gain. We keep you up to date as soon as we have more feedback.

Update 8 – 16/01 10:52
Meanwhile our technical team contacted our hardware supplier in order to find the possible cause and solution.  We keep you up to date as soon as we have more feedback.

Update 7 – 15/01 20:15
Our technical team had to bring webmail off line in order to continue to search for a solution for our e-mail delays. We keep you up to date as soon as we have more feedback.

Update 6 – 15/01 17:04
Our technical team is working non-stop to solve the mail issue. There are several problems causing e-mail to slow down. Our technicians are excluding several possible issues one by one. Every time we get one step closer to a solution. As soon as we have more feedback we will keep you informed.

Update 5 – 15/01 11:03
The scheduled maintenance of this night did not completely resolve the issues, therefore we need to schedule another maintenance as soon as possible. Our mail systems will go completely offline at 12:00 for 20 to 25 minutes.

Update 4 – 14/01 14:39
The data has been rebuilt but our systems are still causing time-outs. Our network team will perform a maintenance tonight (Starting at 01:00) to resolve the issues. During this maintenance you will not be able to receive any e-mails. We expect to resume full service by tomorrow morning.

We can confirm that no data is or will be lost.

Update 3 – 13/01 13:13
We expect that the delay for receiving email messages will be solved within 48 Hrs. The delay is caused also due to the high load during working hours. Edpnet is doing everything to speed up ongoing processes 

Update 2 – 09/01 10:30
Our data storage system experienced a big failure. No data is lost and everything kept on running. The system is currently rebuilding a lot of data and responds slow. You can experience time-out when retrieving your e-mail. Please try again after a few minutes if you have a time-out. Completely resolving this issue will take some days.

Update 1 – 14:43
The issues were temporary resolved but have returned, our network team is still looking in these issues. We thank you for your patience.

Due to a failure on our e-mail system we have to perform an unforeseen maintenance right now. You won’t lose any mails. Please be patient if your e-mail isn’t working at the moment.

Thank you for your understanding.
 
Edpnet apologizes for any inconveniences.


20 January 2014

There is an outage on our network for the region of Antwerp & Limburg. All services of edpnet are down. We are currently investigating the origin of the problem. We will give you more feedback soon.

Update 12:25
Due to a power outage in our co-location for region of Antwerp, connections for Antwerp, Limburg and parts of Flemish Brabant en East-Flanders were disrupted. These connections came back online when the power was restored.

Our apologies for the inconvenience.


10 January 2014

We had an outage this morning between 2AM and 3AM in the Antwerp Area.
As a consequence all user in this region were temporary disconnected.

The cause of this issue was a power failure at the remote co-location of Belgacom. Both of our redundant feeds were impacted.
The cause of the failure is under investigation in dialogue with our supplier.

All services were restored automatically.
If you have no internet after the outage we advise you to restart your modem first.

We apologize for the inconvenience.


09 Januari 2014

Vanwege het falen van onze opslagsystemen worden mails momenteel vertraagd verzonden. Er is geen data verloren gegaan en de systemen draaien nog steeds.De vertraging wordt veroorzaakt door het heropbouwen van onze systemen. Het is mogelijk dat u ook tijdelijk een foutmelding krijgt, gelieve het dan enkele minuten later nogmaals te proberen.

Het volledig oplossen van deze storing kan tot enkele dagen in beslag nemen.

Update 9 – 20/01 16:00
Na werken van dit weekend lijkt het erop dat we de oorzaak van de vertraagde mails hebben gevonden. Door het vervangen van een defecte controller op onze hoofd opslagserver is de snelheid van het systeem drastisch verhoogd. Maandagochtend was de eerste stresstest van ons systeem door u (de klanten) aangezien een maandagochtend  één van onze drukste dagen is voor onze mailserver. De responstijden blijven normaal. We blijven alles monitoren.

Update 8 – 18/01 10:29
Gisteravond hebben we nog een probleem gevonden met een opslag systeem, we hebben dit volledig vervangen, en dit toestel is ondertussen data aan het repliceren, dit zou vandaag afgerond moeten zijn. We hebben ook een defecte disk controller dienen te vervangen in een ander opslagsysteem, deze nacht is deze nieuwe controller opnieuw optimaal beginnen werken. Omdat in het weekend en ‘s nachts de load sowieso lager is, zou alles nu optimaal moeten werken. We dienen tot maandag te wachten of het probleem terug zou opduiken. Onze excuses voor de beperkte communicatie gisteren, we waren hard aan het werk, maar hadden geen updates.

Update 7 – 17/01 11:16
De monitoring van de laatste implementatie heeft afgelopen nacht bevestigd dat we nog niet het gewenste resultaat hebben. Daarom wordt nu, om 11u, nog een volgend onderhoud uitgevoerd. Van zodra wij hierover feedback hebben, wordt u op de hoogte gebracht. Intussen is webmail terug beschikbaar. Mocht u nog op de storingen-pagina terecht komen, gelieve dan even uw browser cache te verwijderen.

Update 6 – 16/01 16:40
In overleg met onze hardwareleverancier werd een mogelijke oplossing uitgewerkt.  Deze wordt op dit moment geïmplementeerd.  In de loop van de avond en de nacht wordt verder opgevolgd om te zien of dit de nodige snelheidswinst geeft zodat er geen vertragingen meer optreden.

Update 5 – 16/01 10:59
Intussen heeft ons technisch team contact met de hardware leverancier en zoeken ze samen naar de mogelijke oorzaak van het probleem.   Wij houden u verder op de hoogte van zodra er feedback is.

Update 4 – 15/01 17:02
Ons tech team is nog steeds volop bezig, het is niet een welbepaald duidelijk probleem dat de vertraging in de mails veroorzaakt. Ze zoeken stap voor stap uit door te testen en de resultaten van de ingrepen te controleren. Telkens komen we een stap dichter bij de oplossing. Wij informeren u van zodra we meer nieuws hebben.

We willen nog even bevestigen dat er geen data is of zal verloren gaan.

Update 3 – 15/01 11:22
Het onderhoud van deze nacht heeft de problemen nog niet volledig verholpen, daarom dienen wij nogmaals een onderhoud in te plannen. Deze zal om 12:00 starten en er wordt geschat dat er gedurende een half uur geen mails zullen worden uitgestuurd.

Update 2 – 14/01 14:42
Onze systemen zijn momenteel terug opgebouwd maar er zijn nog blijvende problemen met het vertraagd ontvangen van berichten. Daarom plant ons netwerkteam deze nacht om 1 uur een onderhoud in. Gedurende dit onderhoud is het niet mogelijk om berichten te ontvangen. We verwachten dat alle systemen terug normaal draaien tegen de ochtend.

Update 1 – 13/01 13:22We verwachten dat de vertraging tijdens het ontvangen van berichten zal worden opgelost binnen de 48 uren. De vertraging wordt mede veroorzaakt door een te hoge belasting tijdens de werkuren. Edpnet doet er alles aan om dit proces te versnellen.

Edpnet excuseert zich voor het ongemak.


26 december 2013

Er is een onderhoud gepland om de kwaliteit van ons netwerk te verbeteren. Gedurende het onderhoud zullen alle edpnet diensten tijdelijk onbeschikbaar zijn (DSL, VOIP, hosting, mail,…).Tijdspanne: 02:00 – 04:00
Downtime: ~60 minutes Het is aan te raden om het modem te herstarten bij problemen na de geplande werken.

Edpnet verontschuldigt zich voor dit ongemak.


26 December 2013

There is a maintenance scheduled to improve the quality of our network. During the maintenance all edpnet services (DSL, VoIP, hosting, mail, …) will be impacted.

Maintenance window: 02:00 – 04:00
Downtime: ~60 minutes We advice you to restart your modem/router after the maintenance has been completed. 
Edpnet apologizes for any inconveniences.

18 November 2013

There was a short outage on our network for the regions East- and West-Flanders, this afternoon at 13h35 until 13h50. Since last night the old line (or our former supplier) has an outage due to a broken network card. No problem for our customers, as our fiber backup took over and all internet traffic was redirected.

This afternoon the old supplier replaced the broken card. Our new Cisco backup equipment detected this hardware replacement and made a wrong reaction on the backup/reroute. Our NOC team is investigating with Cisco engineers why the new equipment made this ‘wrong’ detection.

Meanwhile to avoid this, our NOC team deactivated manually the old line/former supplier. In this case further interventions/changes on the former suppliers network can not intervene our backup/reroute.

Our apologies for the inconvenience.


16 October 2013

There is currently an outage in East- and West-Flanders impacting internet and telephony services. Edpnet is trying to solve this as fast as possible.

We’ll let you know as soon as any ETR is available. Sorry for the inconveniences caused.

Update 16/10/2013 12:54: Our local supplier is working on identifying the problem. We’ll keep you informed of any new information on this matter. Sorry again for the inconveniences.

Update 16/10/2013 13:05: The fault is detected: there’s a major fiber cut between Gent and Brussels. No ETR available yet, we’re cheching it with our supplier.

Update 16/10/2013 13:43: The fiber cut is finally located. We expect the splicers to arrive on location at 14:15. We’ll keep you up to date with the progress of the works and we’ll let you know the final ETR once it’s available. We apologize again for the inconvenience and thank you for your understanding.

Update 16/10/2013 14:48: The splicers are working on the issue now. Unfortunately, we still cannot provide you with a confirmed repair time. We’re monitoring the situation and will update you soon.

Update 16/10/2013 15:40: The problem seems to be quite complicated to solve, there’s almost a hundred of fiber pairs to be spliced. The ETR for the moment is around 18:00. We’ll let you know once it’s confirmed.

Update 16/10/2013 17:10: The splicing works are still ongoing. The ETR remains the same, we expect the issue to be solved at around 18:00. We’ll keep you posted.

Update 16/10/2013 18:24: The splicing works are still ongoing, got confirmation that first fibers are getting online, ours isn’t online yet. We’ll keep you posted.

Update 16/10/2013 19:42: Everything seems to be up and running again!

Thanks to everybody for your patience and understanding, we hope our next blog post will be about the new fibers being up!


19 September 2013

There is currently an outage in Ghent. It is impacting all of the internet connections in East and West Flanders. We expect the issue to be solved during the morning .

Your internet connection will re-establish itself automatically or after a reboot as soon as the problem is solved.

Update 8:40
Problem located, local Belgacom interconnect problem, no fiber outage, Belgacom ticket has been raised

Update 8:48
Belgacom acknowledges that the outage is at their side, some other clients of them are impacted. ETR for the moment is 10:00

Update 9:47
Our customers let us know they are able to surf again. Problems should now be resolved. If you can not get online, please restart your modem first.

Our apologies for the inconvenience.


8 September

There was a small interruption towards the our east- and west-flanders DSL customers, we are investigating this issue.

According to our network team, this issue should now be resolved.
We apologize for the inconvenience.