Tonight (2/22/10) around 6PM Eastern we will begin to implement split GALs on the Australia Exchange 2007 server. Between 6PM and 8PM Eastern there is expected to be brief interruptions in service as the new security policies are implemented.
The Ahsay service on backup74 has been temporarily stopped as we migrate users to the newer volumes. Service is expected to be restored by 4pm Eastern.
The Dewey BES service has expierenced a few random issues since Microsoft patching on the 10th. In an attempt to revert the changes, we’re in the process of removing the updates before restoring from a backup. Service on DEWEY BES will not be interrupted throughout the extended maintenance of the BES server. Update 10:50 AM Eastern: The BES service for Dewey has been restored 100%. We found an issue with two security updates, including KB977914
The Ahsay OBS service on backup74 is currently stopped as we migrate users to larger volumes. Service is expected to be restored by our peak backup time of 12 am Eastern.
At 8 PM Eastern tonight we will be taking backup74 offline to reattempt storage installation. The installation and provisioning is expected to take up an hour and a half and service will be automatically restored after the upgrade is complete. Update 10:07 PM Eastern: The new controller install failed again. We’ve hooked up a temporary drive to alleviate the space issue while we order a new controller. Service is expected to be restored by 11 PM Eastern. Update 10:58 PM Eastern: Service has been restored.
We’ve had reports from a few clients that they are having issues accessing our Exchange 2007 server DEWEY. We are currently investigating into this issue. Update 11:20 AM: Service on DEWEY has been running 100% over an hour. It seems close to 10 PM Eastern on 2/4/2010, IIS had issues compiling the metadatabase..which ultimately did not throw any errors in windows logs or in any of our monitoring software. We ran a quick repair on the metadatabase soon after being notified of service outage and restarted IIS which resolved the issue.
Backup74 will be going offline from 6pm Eastern till 10pm Eastern today (2/2/10) for extended maintenance. We will be installing 4TB of additional storage to the server. Update 8:20 PM Eastern: After installing the new controller the server was unable to boot property. We’ve removed the new controller and ordered a new replacement.
We have received reports from a few partners that ExchangeDefender users are not receiving mail released from Quarantine. We are investigating into this issue and assure that no mail has been lost. If you experience issues in releasing mail from quarantine please open a support ticket at support.ownwebnow.com including the senders email address , recipients email address, and time stamp. Update – Noon EST – We have resolved the issue that caused the release mechanism to fail, the problem has been fixed and as of about 11:30 the full functionality has been restored. Now a word from Vlad Mazek, CEO:
Throughout the day the ExchangeDefender outbound grid has been fighting extremely large mail queues and hour long delivery delays. The source has been identified as a DDoS attack and we’ve taken all mesures to remove the mail. Legitimate mail that hasn’t been delivered will be delivered throughout the next couple of hours. We highly apologize and we are making modifications to the outbound grid throughtout the next limit to prevent flooding.
We will be holding an extended maintenance window this weekend affecting admin.exchangedefender.com systems:
During this time window access to admin.exchangedefender.com will be intermittent as we undergo a major networking and hardware update to handle the expansion and additional services. We will start posting updates to this blog during the 4 AM – 7 AM.
Our second US OBS server, backup74 will be going offline for the next three hours as we begin to migrate users around different volumes. Service is expected to be restored before 5 PM Eastern.
The Offsite Backup Server, backup74, will be going offline at 3:00 PM Eastern to install hotfixes to OBS. The server is expected to be offline for 30 minutes and service should be restored by 3:30 PM Eastern. Update 3:01 PM Eastern: Service has been disabled on backup74 as we begin patching. Update 3:11 PM Eastern: Service has been restored on backup74 and the server is now running version 5.5.5.5-2.
At 4:10 PM Eastern we will be rebooting DEWEY for feature enhancements and more analytical monitoring. The reboot is expected to last no longer than 10 minutes and service should be 100% functional by 4:20 PM Eastern.
We are beginning a maintenance schedule for backup74.ownwebnow.com. During the maintenance schedule, access to backup74.ownwebnow.com will be interrupted, however, service is expected to be restored by 12:00PM Eastern. Updated 1:30 PM Eastern: User migration is taking a bit longer than expected. The final user move is in progress and the server should be online before 3pm Eastern. Updated 2:50 PM Eastern: User migration has completed and service has been restored to backup74. maintenance
We are about to reboot huey.exchangedefender.com due to user accessibility complaints. Service is expected to be fully impacted on HUEY for the 15 minutes while the reboot commences. Update 3:21 PM Eastern: The server is back online from the reboot and service has been restored.
We are in the process of replacing the certificate for HUEY in our Exchange 2007 cluster. We highly apologize for the inconvenience however service should be restored before 3pm Eastern. Updated 2:42 PM Eastern: The certificate has been successfully replaced on HUEY. Service on HUEY is now 100% operational.
Last week (January 8th and 9th) we received a dozen reports of messages that simply vanished in the ExchangeDefender system. Upon investigation it turned out that one of the antivirus engines was picking up false positives: marking messages with certain PDF attachments as infected when in fact there was no infection there. The actual infection was simply a detection of an exploit, one that can easily and inadvertently be created by older versions of Acrobat. We have removed the antivirus engine from the rotation (don’t worry, everything is still being scanned by several other scanners). While the problem in the definition files was already addressed (Exploit.PDF-9669) and widely blogged and discussed, we need a way to deal with false positives. Prior to this we have never had an instance of a reported false positive with an antivirus engine but as more antivirus vendors get into the business of not just detecting viruses and worms but also exploits and other dangerous content, our reporting will have to get better as well. The bigger question here is: Why was I not notified? If this happened here, it would also explain why I am never received any of the other messages. Allow me to address that in two ways: 1) Almost all of our “missing messages” tickets are related to the messages being quarantined as SPAM and not coming into LiveArchive. At the present time there is no way to get a SPAM message into LiveArchive, even after it’s released from the Quarantine. Because our replication is done at the scan time, we have to move the copying protocol elsewhere to enable post-release and SPAM content.
2) We have never before seen a false positive from an antivirus engine. We’ve seen it crash, we’ve seen it fail to detect a real infection, we’ve seen it bring the scanning node to a crawl and just about everything you’d expect from a piece of security software: just never a false reading. Consequently, we never wrote a process to monitor for the false positives and we never bothered to present the infection logs because so many contained meaningless junk. Several years ago, after countless alerts for Sober and Nimda and so on, we disabled end user reports for antivirus and it was eventually dropped from the product completely.
IMPORTANT: While these infections appeared to be lost forever, we do have them stored on our servers. Reported messages are being released (by hand) by our support teams so if you know the message sender/recipient/subject and date the message was sent, we can retrieve the message and deliver it. -Vlad
We’ve had reports from a few partners that RPC over HTTPs is not connecting on HUEY. We are rebooting the server to rectify the issue. Update 10:12 AM Eastern: Service has been restored and we have confirmed RPC over HTTP is functioning properly.
We are currently working on livearchive.exchangedefender.com and the server is currently offline. The IIS application pool keeps crashing since a Windows Update from last night. We expect to have the server up shortly. Update 9:00 AM Eastern: We’ve resolved the issue with livearchive OWA.
All of our BES servers are currently offline as we move the virtual disks to the new RAID set added in yesterday. All servers are expected to be online in the next 45 minutes. |
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
