Midland Computers NOC
Planned maintenance Thursday morning

We will be undertaking essential maintenance between 06:00 and 08:00 BST tomorrow morning (Thursday 19th April).
This will mean a complete outage of all ISP services at a point between those hours, (06:15 target time)
As usual we will aim to keep disruption to a minimum and anticipate a total outage to last approx 10 mins in the allotted time slot if everything goes to plan. The extra time is scheduled for testing and roll back if necessary.
Update: Connectivity has resumed as normal.

Emails rebounding?

We have received a number of notifications that email being sent to alias addresses are rebounding as “unknown user” from server “homer”.

Our engineers are investigating this issue and will update when we know more.

UPDATE

We believe the issue to now be resolved

Server Crash

Approximatly 2 hours ago one of our mail servers crashed, meaning that some customers would not have been able to retrieve or send emails. The issue has now been rectified and the server is performing as expected. The desgin of the email platform means that no emails would have been lost during this fault and should you experiance any further issues please contact our support team via the usual manner.

A few interesting stats…

We’ve just been reviewing some of the mountains of analytical information that we handle and observe to spot trends, potential problems and plan for growth.

Here were some figures you might find interesting.

  • Last week our email cluster processed 6.1Gb of emails in a single day.
  • Since the 14th of July 2011 (when the cluster was initialised) it has processed in excess of 3,136,910 emails or approximately 360gb.
  • Our gateway spam protection prevented 15 million spam emails from being processed by the cluster. 
  • That means 79% of incoming email traffic was spam!
  • Of the 21% of email that was processed, approx 10% of that was identified as being spam by our heuristics detection.
  • Less than 1% of of processed emails contained a virus.
  • We experienced just 2 false positives out of over 3 million! (A false positive is a genuine email stopped by mistake). 

Some quite amazing figures there that illustrate just how much traffic we deal with and the effectiveness of our email cluster service.

BT ADSL Outage

We are aware that a number of customers who have BT ADSL connections are currently experiencing an outage. If your broadband is provided by BT this will affect all services provided by Midland Computers.

We are currently storing and queuing email for our customers who subscribe to our Advanced Hosting package with SMTP delivery (Microsoft Exchange) .

If you would like to know more about our Advanced Hosting package and how it can protect your business at times like these please contact your account manager on 01952 588688

Connectivity Issues

We are aware of an issue where certain customers are unable to view websites hosted with Midland Computers. We believe the issue is down to an issue within our providers network. At present we are unable to contact them as their phone lines appear down also.

We will update the site again once we have further information


Connectivity appears to be restored, we are continuing to monitor and await feedback from our provider.

[update] We have this morning received the following update from our supplier:

“We have suffered a major outage impacting services in parts of the country. Services are now restored and investigations to determine the root cause are ongoing. There will be further information to follow as and when available.”

[update] We have received the final RFO from our supplier:

“On Tuesday evening (17th January), Virgin Media Business experienced a network outage that impacted customers using IPVPN, Managed Internet, National Ethernet and broadband services. The problem was quickly traced to a failed cooling unit that had caused a core backbone router to overheat. Due to exceptional circumstances, other backup equipment also failed to activate properly. We’re are now working through a thorough investigation as to why this happened. Engineers resolved the fault in about an hour. Virgin Media Business would like to apologise to all customers impacted by this outage.”

Problems sending to Hotmail

We are aware of an issue for some users sending emails to Hotmail users. We are currently looking into the issue and will update the post when we have an update.

[update] - We have rerouted the email traffic through an alternative server to resolve the issue

Primary circuit failure

Our primary connect has just failed, the backup lines have switched over as automatically designed to in this situation.

Please be advised there maybe a slight performance decrease of services at this time.

We are investigating the issue with our providers and hope to have an update soon.

[update] - the primary circuit has returned to normal service. We are continuing to investigate the reason for this failure.

[update2] - we have received notification that our suppliers core routers experienced some instability affecting all attached client services. They are investigating the reasons for this.

[update3] - We have been advised that a power outage in Wolverhampton caused the outage and that we should not expect further disruption.

Email issues weekend 22/23 October

We have been experiencing email problems this weekend with our legacy email server.

On Saturday morning we received support reports relating to Alias email addresses were bouncing back to the sender as ‘unknown address’. Default main email addresses were unaffected.

An alias email address is also described as a forwarding address. For instance if your normal email address is bob@mydomain.com, but you also have info@mydomain.com forwarding to bob@mydomain.com then you will not have received the info@mydomain.com email. The sender will have received a notification of an issue.

Unfortunately this error took considerable time and effort to resolve but was fixed at approx. 22:00 Sat night.

On Sunday morning further reports came in regarding some emails not arriving. Investigation of this identified a problem with another area of the email pipeline. This has also been resolved and any email not received will automatically be resent.

We are speeding up our migration of all email users to our new clustered email solution which will render this older and increasingly troublesome server obsolete.

We are continuing to monitor the situation.

Windows shared hosting security update

We are conducting an urgent security update on our Windows shared hosting platform. If your website is on this platform you may experience approximately 10 mins downtime between 11:00 and 12:00 today (Mon 10th October)

Edit: Work complete with very minor downtime upon reboot.