Issue mailserver

Dear customers,

We are currently investigating issues with our mailservers. These are estimated to be resolved in 5-10 minutes.

Update 10:40 AM: There was an issue with the connection between the storage servers and the mailservers. Everything is back in service now.

We apologize for the inconvenience

The edpnet team

*All times are listed in CET, Central European Time


13 November – Speed issues in Antwerp, Limburg and Flemish Brabant

Update: Services are restored, there was a cabling issue in the collocation center.

Dear customers,

One of the access lines between the Proximus network and our network is currently down. As a result of this outage, all traffic is automatically rerouted.
Customers in the regions Antwerp, Limburg, and parts of Flemish Brabant may encounter slower internet speeds (packet loss and a higher latency) during this rerouting.

A Proximus technician will go onsite to resolve the issue.

Description: 
Start: 2014-11-13 09:04 AM*
End: 2014-11-13 11:19 AM*
Impacted services: Internet services in the above mentioned regions.

We apologize for the inconvenience

The edpnet team

*All times are listed in CET, Central European Time


23 October – Packet-loss due to DDOS attacks

Dear customers,

Edpnet has experienced a massive DDOS attack on several parts on its network. This has caused a lot of its links to experience heavy packet-loss. Everything has been redirected to reduce further impact. All traffic has been normalized.

Description:
Start: 2014-10-22 22:40 PM
End: 2014-10-23 Between 07:00-08:00 PM (Towards Amsterdam & Russia)
End: 2014-10-23 08:20 PM (Traffic towards Brussels)
Impacted services: All traffic may have experienced packet-loss

(Dates have been corrected)

We apologize for the inconvenience

The edpnet team


17 October – VOIP Switch fail

Update 14:07 : The issue has been resolved.

Dear customers,

One of our VoIP switches has failed. The backup switch has taken over. If you are using a trunk, please prioritize calls via 212.71.16.196. If you are a Cloud PBX customer, please call our helpdesk via 03 265 67 00 to do this for you.

Description: VoIP switch failure
Start: 2014-10-17 13:55 PM
End: 2014-10-17 14:07 PM
Impacted services: edpnet SIP Trunks (Please use the backup server)

We appologize for the inconvenience

The edpnet team



4 April 2014

Dear customers,

Our DNS servers are currently being used in a DNS amplification attack. Our NOC is currently doing its best to block off as much as possible.

You may have issues reaching some websites during this attack. It’s best to retry again when a lookup fails.

We apologize for the inconvenience.


19 February 2014 – Telephony issue

Update 15:25
We have restarted our systems, everything is back up and running. We are looking into the cause with our supplier.

Dear customers,

Since 2:30 we are having a problem with Voip and SIP trunks, including our own telephone systems. We are currently not or difficult to reach via telephone.

Our network team is currently trying to resolve this issue as soon as possible.

More news will follow.


17 Februari 2014


Update 18/02/2014 – 14u00
Spamhaus reageert (nog) niet op onze tickets en mails, maar ze reageerde deze middag wel op Twitter. Wij hebben momenteel nog geen officiële bevestiging via email van Spamhaus dat we gedeblokkeerd zijn, maar het lijkt er op dat onze range is vrijgegeven. We zien online ook op http://www.spamhaus.org/sbl/latest/ dat we een GO hebben gekregen. Enkele van onze eerste testmails lijken te lukken, andere dan weer (nog) niet. Wij blijven verder testen en wij adviseren u om dat ook te doen. Het kan enkele uren duren vooraleer het internet volledig up to date is van het vrijgeven van onze IP range. Van zodra Spamhaus naar ons officieel communiceert, zullen wij onze blog terug een update geven.

Update 18/02/2014 – 10u30

1 spam listing/melding dient door Spamhaus nog bevestigd te worden. Na bevestiging zal Spamhaus (van zodra zij dat effectief ook willen doen) onze volledig IP range deblokkeren. Zoals reeds gezegd, blokkeert Spamhaus volledige IP ranges van providers in plaats van het specifieke range of IP-adres van de spammer zelf. Blokkering van een volledige IP range van een ISP is een zeer drastische maatregel, maar als een onafhankelijke organisatie kunnen zij zelf de regels en procedures bepalen. Hierdoor hebben zij vaak providers soms ongewild in een wurggreep. Desondanks hebben wij steeds een goede communicatie/verstandhouding met Spamhaus. Onze excuses voor de ongemakken.

Update 17/02/2014 – 16h47
12 individuele Spamhaus vermeldingen zijn geblokkeerd. 8 daarvan zijn bevestigd door Spamhaus. We wachten voor de laatste 4 vermeldingen om door Spamhaus erkend te worden waarna we hopen dat Spamhaus onze volledige reeks zal deblokkeren.

Beste klant

Spamhaus.org heeft onze IP ranges geblokkeerd omwille van 2 klanten die blijkbaar spam uitstuurden. Intussen heeft ons Network Operations Team beide klanten zelf geblokkeerd. Momenteel zal u problemen ondervinden in het versturen van emails naar personen die Spamhaus.org gebruiken als spamfilter.

Enige tijd geleden hadden we gecommuniceerd dat Spamhaus.org steeds drastische maatregelen neemt, nl. het blokkeren van volledige IP-ranges ipv de specifieke spammende IP-adressen.

Als ISP willen we trafiek niet filteren en laten we onze klanten met vast IP-adres toe om hun eigen mail servers te hebben. We kunnen proberen om op een pro-actieve manier trafiek te checken op mogelijk misbruik, en we hebben nze policy ivm trafiek-filtering reeds verscherpt omwille van gelijkaardige problemen in het verleden. Bovendien hebben we poort 25 gesloten voor onze klanten met dynamisch IP-adres om misbruik en spam te voorkomen. Edpnet implementeerde zelfs een Cisco Iron Port mail systeem om spam tot het minimum te herleiden.

Ons Network Operations Team is in contact met Spamhaus.org om onze IP-ranges zo snel als mogelijk ge-deblokkeerd te krijgen.


17 February 2014

Update 18/02/2014 – 13h45
While not responding to tickets or mails from our NOC team, Spamhaus did respond on one of our Tweets. We didn’t get an official confirmation via email from Spamhaus, but it seems that our range isn’t listed anymore. We can see that we had green light on http://www.spamhaus.org/sbl/latest/ We will do some testings and we advize you to do this as well. It can take a couple of hours before the entire internet is up to date of the deblocking of our range. From the moment that Spamhaus confirmed to us officially, we will update our blog again.

Update 18/02/2014 – 10h15

1 listing remains which needs to be acknowledged by Spamhaus. (After acknowledgement Spamhaus will deblock our IP range as soon as they are willing to do so). As said before, Spamhaus blocks entire IP ranges of ISP, instead of 1 specific IP address or range of the spammer. This is a drastic decision, but as an independant organisation, they set their own rules and procedures. And in this way they often have internet service providers in an unwanted stranglehold. Nevertheless we always have good communication with Spamhaus. We are very sorry for the inconveniences.

Update 17/02/2014 – 16h47
12 individual spamhaus listings are blocked. 8 are acknowledged by Spamhaus. We are waiting for the last 4 listings remaining. After we surely hope Spamhaus can deblock our entire IP range.

Dear customer,

Spamhaus.org listed our IP ranges, due to 2 customers who seems to be spammers. Meanwhile our NOC team took the necessary steps to block these spammers, but you are unable to send mails to people who use spamhaus.org as a filter to receive e-mail.

As we already communicated in the past, spamhaus.org always take extreme measurments by blocking entire IP ranges, instead of the one specific spamming IP address.

As an ISP we do not do any filtering of traffic and we do allow fixed IP users to have their own mail servers, but we try to pro-actively scan as much traffic as we can for possible abuse. We already redefined our policy regarding traffic filtering to prevent such issues, but it seems now that we need to protect ourselves and our customers even more.

We still believe that over the years we already took quite a lot of measurements to get the internet free from abuse and spam as we closed port 25 for all our dynamic IP users and we implemented a state of the art Cisco Iron Port mail system.
Nevertheless we still do not do any traffic filtering and we don’t want to.

We are in contact with Spamhaus.org in order to delist our IP-ranges as soon as possible.


8 January 2014 – Mail Exchange Issue

Maintenance to improve stability.

Update 13 – 21/01 17:03
We schedule a maintenance, tomorrow morning at 05:00, to improve the performance and stability of our mail servers. During the maintenance all mail service will be temporary unavailable. No mail will be lost.

Maintenance will start on 22/01 around 05:00 and will take about 30 minutes.

Update 12 – 20/01 10:30
After the works of this weekend it seems that we have found the origin of the slow responsiveness of the mail servers. By replacing the defect disk controller in our main storage server the speed of our mail servers increased drastically. Monday morning we had our first real stress test by you as customer (Monday morning is the most busy time frame in the week for our mail servers) and the response time of our mail service stayed normal. We keep on monitoring our mail system but we presume this issue finally is solved.

Update 11 – 18/01 10:29
Yesterday evening we found another problem with the storage systems, we completely replaced one of the storage systems, this device is now replicating  data, and should be finished today. We also needed to replace a defective disk controller in another storage system, this night this controller started working as it should again. Due to the fact that the load is lower in the weekend and at night, everything should anyway work optimal. We need to wait until Monday to see if the problem rises again.  We are very sorry for the lack of communication yesterday, we were working very hard, but had no updates.

Update 10 – 17/01 10:57

The overnight monitoring didn’t give the expected result. Therefore our technical team will perform another maintenance at 11h. We keep you up to date as soon as we have more feedback. For your information: meanwhile webmail is back online. Please clear your browser cache if you still arrive on the issues page when connecting to webmail.

Update 9 – 16/01 16:40
Together with our software supplier we are currently implementing a new possible solution. During the next hours and night we will monitor this new implementation to see if it gives the necessary speed gain. We keep you up to date as soon as we have more feedback.

Update 8 – 16/01 10:52
Meanwhile our technical team contacted our hardware supplier in order to find the possible cause and solution.  We keep you up to date as soon as we have more feedback.

Update 7 – 15/01 20:15
Our technical team had to bring webmail off line in order to continue to search for a solution for our e-mail delays. We keep you up to date as soon as we have more feedback.

Update 6 – 15/01 17:04
Our technical team is working non-stop to solve the mail issue. There are several problems causing e-mail to slow down. Our technicians are excluding several possible issues one by one. Every time we get one step closer to a solution. As soon as we have more feedback we will keep you informed.

Update 5 – 15/01 11:03
The scheduled maintenance of this night did not completely resolve the issues, therefore we need to schedule another maintenance as soon as possible. Our mail systems will go completely offline at 12:00 for 20 to 25 minutes.

Update 4 – 14/01 14:39
The data has been rebuilt but our systems are still causing time-outs. Our network team will perform a maintenance tonight (Starting at 01:00) to resolve the issues. During this maintenance you will not be able to receive any e-mails. We expect to resume full service by tomorrow morning.

We can confirm that no data is or will be lost.

Update 3 – 13/01 13:13
We expect that the delay for receiving email messages will be solved within 48 Hrs. The delay is caused also due to the high load during working hours. Edpnet is doing everything to speed up ongoing processes 

Update 2 – 09/01 10:30
Our data storage system experienced a big failure. No data is lost and everything kept on running. The system is currently rebuilding a lot of data and responds slow. You can experience time-out when retrieving your e-mail. Please try again after a few minutes if you have a time-out. Completely resolving this issue will take some days.

Update 1 – 14:43
The issues were temporary resolved but have returned, our network team is still looking in these issues. We thank you for your patience.

Due to a failure on our e-mail system we have to perform an unforeseen maintenance right now. You won’t lose any mails. Please be patient if your e-mail isn’t working at the moment.

Thank you for your understanding.
 
Edpnet apologizes for any inconveniences.