We have restored the service to the small portion of users that were affected by its outage this morning. The issue had to do with a hotfix provided by Microsoft. Hope everyone in the midwest and Ohio weathers through the storms that have knocked much of the power out in that region.
We must have angered the Internet gods because this Monday has been nothing short of tremendously disappointing. Pictured below is my staff working on the issues:
On to the specifics: ExchangeDefender reports did not run last night and will likely remain offline until close of business today. We have had two switch crashes on our load balancers in front of our shared mail1 and www1 hosting services. Our offsite backup upgrade does not seem to be validating the certificate requests so https:// requests are failing (http:// still works fine, and data is encrypted on the client side so the transport mechanism isn’t as relevant - but if you’ve set https:// your backups are failing so we are treating this as a very serious issue) Somehow, the roof is still above us and we have power. For now. All the outstanding issues are being filtered through by my teams and will have service restored to 100% across the entire product portfolio - by the end of business today. Update: As of 5 PM EST the ExchangeDefender reporting is back online, all the network issues have been resolved. The Offsite Backup service is still available via http:// but we are still working with AhSay to get the certificate issue resolved. Will update further on this as soon as I have more information. Update: As of 11 PM EST all offsite backup grids now respond with the valid SSL certificates on the SSL port. Looks like the ugly Monday is finally behind us. Sincerely, Vlad Mazek, CEO
We have received several reports this morning about our IP address blocks being on Verizon’s RBL. The following errors were given to some of our customers on ExchangeDefender:
In our calls and discussions with Verizon we have received a confirmation that we are not and have not been on their RBL. At this point the mail is routing correctly so we are just chalking this up to there being a temporary glitch with Verizon’s RBL systems.
Over the past two days that the reports service has been restored we’ve discovered a few bugs in the system that prevented proper delivery and branding of the SPAM reports. Even though they were generated properly, the reports got routed through the ExchangeDefender inbound network instead of direct to the servers. This unfortunately may have gotten trapped in the junk mail again. This issue was corrected at 11 PM EST (4 AM GMT).
We are talking advantage of an extended holiday weekend in United States to perform network upgrades and maintenance as well as a software rollout on our email reporting grid for ExchangeDefender. We have rolled in ExchangeDefender 4.0 upgrades to this system and are taking an extra day to put it through it’s paces and make sure it’s 100% solid. For our customers abroad that will be affected by the email reports please keep in mind that this legacy system is just one of the ways to access junk. The recommended and preferred way of accessing SPAM quarantines for ExchangeDefender is the web portal at https://admin.exchangedefender.com and we also offer the SPAM Monitor desktop software with hourly alerts. We anticipate regular daily reports to resume on Tuesday.
Good morning - we are currently investigating two blacklists from large ISPs targeting one IP address on the ExchangeDefender network. Two services are identified as Verizon and FrontBridge and we have opened requests to be removed from both along with any data that might help us find out why the lists were put in place to begin with given our 0 tolerance for SPAM. The rejections are marked by:
In the meantime, you can change your outbound smarthost to outbound1.exchangedefender.com if you experience the problems above. On the funny side: We find it hilarious that Microsoft is linking to their own KB articles in the rejection note for a problem that is caused on their own servers - how about something more helpful like postmaster or delisting contact address or URL! Even more surprising is that Microsoft FrontBridge is running on an open source Postfix mail platform, not a Microsoft one. We will update you on the delisting process as we get more information. The problem should not impact many senders as 65.99.255.232 is just one of the nodes in our outbound network.
Earlier today we completed the rollout of 450 new servers to the ExchangeDefender family all over our American network. The introduction and initial sync of the new nodes did allow some junk through as well as introduce a slight today (maximum reported 1 hour from one system that nearly immediately went into maintenance mode) but as of roughly 11:30 AM EST all is good. Additional 600 nodes are planned in our global expansion leading up to ExchangeDefender 4.0 launch. We are also looking at additional data centers on both coasts at the moment scheduled to go live this fall. Update: 2:24 PM EST: We are happy to report that all the nodes have now converged in the scanning network and the SPAM filtering is back at its usual levels (and to be tightened up even further later tonight). You may have seen an increase in SPAM over the past few hours while the nodes were joining the network and accepting new programming but you should be seeing far less SPAM going forward.
We have several reports from our UK and Ireland customers of the rise in the amount of junk mail passed through ExchangeDefender this morning. Aside from a strain of CNN-forged SPAM we are not seeing any issues in ExchangeDefender nor do our stats show anything out of the ordinary at the moment. We are investigating the situation. The SPAM regarding CNN is already in the filters and should be stopped going through further. For anything else that may slip through please forward the message with SMTP headers to spam@ownwebnow.com and we will gladly investigate it. Update: We had a rule update that unfortunately offsite all the other CNN rules and let that junk through. The team is now filtering it through both the pattern search and hyperlink drop on the domains used to get traffic. We are seeing a few other SPAM strains getting more popular today as well (Wall Street Subscription scam, fake MSN alert to download Internet Explorer 7). All of these are now effectively being filtered by ExchangeDefender which undergoes thousands of updates a day but due to the CNN rules that have been changing a lot over the past few days, and in light of the six complaints we got this morning, we felt it was important to update in more detail than usual. Update 2: We are seeing things under more and more control as we continue to filter out the strains of the three major junk items. As a matter of policy we do not publish our filtering technology or keywords or scores but we are currently tracking the variants of CNN, WSJ, Internet Explorer 7 and a few smaller ones.
As you may be aware, we have two data centers in Los Angeles on Wilshire Blvd. Earlier today, this area suffered a 5.8 magnitude earthquake. No systems were affected, no impact on any power feeds or network connections. Earthquakes tend to be followed by smaller “aftershocks” and we will be updating this post with details of any relevant information that may become available.
Over the weekend we tested and perfected a new method for managing archive embedded dangerous content. During the deployment of the new software some archives were improperly classified as dangerous and archives (.zip, .arj) removed. That issue has been solved as of Sunday evening. As a point of reference, ExchangeDefender does not allow executable attachments (.exe, .bat, .com or .pif) in either standalone or archived mode. That means even if you zip the file up it will be picked up by a scanner. If you zip a zip file, the system will reject to process it. This has been our long standing tradition of not allowing dangerous content through the network because virus scanners sometimes do not react as quickly to the rise in malware and our responsibility is to protect our customers. If you need a dangerous attachment really bad, for the safety of the less IT savvy people in your organization, please try to find alternate means such as a web sharing tool or a freemail account. We have also addressed this need in ExchangeDefender 4.x which is scheduled for August 19th.
Following is our reboot cycle for our global Exchange 2007 network. Microsoft recently published the Exchange Server 2007 SP1 Update Rollup 3, SQL Server Service Pack 2 and several security patches that have been tested by OWN and approved for rollout. Scheduled reboots are as follows:
We do not anticipate any outage or issues with the patches, they have passed lab tests without problems.
We are currently addressing a processing delay in ExchangeDefender antivirus scanning engine. One of our virus engine vendors had distributed a faulty update which has caused a backlog of messages that have been quarantined for further inspection.
In this case the corrupt message was passed on to ExchangeDefender which quarantined messages for further scanning which is far more expensive and processor intensive. We have responded immediately and removed the engine, however, even slight issues can cause huge problems when you process as much mail as we do and it has introduced a slight delay in the processing of messages. The issue started at roughly 3:10 and was resolved by 3:40. At the time of this message we see around 60% of our nodes processing messages within our ordinary SLA (seconds) and we expect the rest of the network to catch up shortly. If you experience any delays, even extensive in nature, it is due to the above problem which will within 30 minutes be completely under control.
We are conducting some routine maintenance on our Windows portion of the Virtual Hosting and Web Hosting network. We are applying patches, installing new hardware, general system maintenance task. All systems should be affected by a brief outage, and will be back within an hour at most. Update: Maintenance cycle completed.
We are currently addressing an issue with Exchange 2007 OWA. You will see the following text when attempting to login:
We will have this addressed momentarily and update this site. Update: 2:45: Problem has been solved.
We are conducting maintenance on our Offsite Backups architecture. 08:00–14:00 EST is our slowest time of the day and we’ll have the systems back online in time for the nightly backups.
Earlier today we had to flush the queues on ExchangeDefender outbound server due to the large number of corrupt queue files sent by one of our customers malfunctioning servers. If your messages were not delivered during the window between 5am - 7 am central (GMT -6) please resend them. The problem has been solved temporarily, but we will be holding an urgent maintenance window this Wednesday, 5/14, to address the core of the problem. P.S. Significant number of servers were backlogged during this process. That mail has been processed without issue.
We are currently tracking accessibility problems on mail1.ownwebnow.com. Please stand by while we research the issue, the server appears up but several customers are reporting access issues, we are trying to resolve them right now. We will update this site as soon as we have more information. The cluster is currently undergoing a reboot. Update: All issues have been resolved, a scheduler service hung on the load balancer.
We are currently investigating a network event in our Dallas region network centers, since approximately 5 AM EST we have been receiving complaints about network connectivity and availability. There are currently no outages and there have been no outages but certain customers are unable to reach the services on our network. If you are experiencing an issue, please open a support request and include a traceroute to the service you are trying to reach (ex: 65.99.255.50). We will update this ticket as soon as we have further information. Update (9:05 AM EST, 14:05 GMT): We believe the network issue some of our customers have experienced has been resolved. Particularly affected were some of our UK customers (not BT) and local customers with Level 3 connectivity. Although we see the traffic back up at usual levels, it may take about an hour or so until all the mail catches up and gets delivered.
We are currently working with RoadRunner (formerly Time Warner, AOL) service provider in United States, they are experiencing issues with their SMTP servers and randomly rejecting SMTP traffic. Currently mail is flowing through but some is bouncing back from them due to a reason they are still trying to narrow down. We will update when we have further information or a resolution. This issue affects our entire global network, and some external sites we have tested. Update: 6:34 PM EST: Even though we have not been officially updated, the problems with RoadRunner appear to have been resolved.
We will be starting the SP1 upgrade on our systems in roughly two hours (10 AM EST, -5 GMT) and expect to have all operations completed by noon. To minimize the surprises and potential conflicts, the entire cluster and all its members will be patched at once. That unfortunately does mean a bit of total outage, but it does minimize the chance of anything breaking in the process. Hosted Exchange has been one of our most solid products and we look forward to keeping it that way. Update: Exchange 2007 Service Pack 1 is now up and running, systems have been imaged and we are all done. |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
