Throughout the night we will be installing new drives to a few LOUIE servers. The following services will be intermittently interrupted throughout the upgrade which is scheduled to last between 5:30 PM and 7:00 PM Exchange: The CAS array host will be rebooted which will leave Outlook disconnected for up to five minutes BES: The BES database is being moved to a new drive which will leave BB devices unable to synchronize for up to 30 minutes Sharepoint: The Sharepoint SQL host is being moved to a new drive which will leave Sharepoint portals unavailable for up to five minutes Update 7:14 PM Eastern: The database move for the BES server is taking longer than expected. We are still working to finalize the upgrades as soon as possible. Update 8:02 PM Eastern: The CAS array is being upgraded now. Users will experience two brief disconnects which are scheduled to last up to 10 minutes each.
Gmail is apparently experiencing technical issues. Several users have reported trouble sending mail to Gmail over the past hour. While we are unable to isolate it at this point we have contacted Gmail and are awaiting response. The issue does not seem to be widespread, only about 10% of messages sent to Gmail terminate with the 5.x.x series error (permanent failure) and it is being issued by Gmail servers, not ExchangeDefender. We have alerted Gmail and will update this blog post when they notify us of any problems being resolved on their end.
The livearchive service has been degraded and new mail is not being delivered to the server. We are in the process of truncating log files and optimizing the storage controller.
You may have noticed an unusual rise in the amount of SPAM getting through to select users on your domain. We have isolated dozens of domains against which large scale SPAM attacks have been raised, typically from trusted (reputable) mail infrastructure. We have been isolating recent outbreak of SPAM that is originating from sources that are typically trusted and managed by larger providers. We have designed new rules, tested them against known sites and our own production domains and we can confirm that we expect the performance as of 2 AM EST to improve significantly. As usual, if you continue to see unusual SPAM levels please use the Outlook 2007 addin and report anything that gets through as the spam@ownwebnow.com alias will be discontinued shortly.
SCROOGE (Exchange 2007) will be going down momentarily for a reboot after tripping our monitoring threshold a few times. The reboot is being done as a proactive measure to avoid any issues during ‘business hours’. The server is expected to only be down for 15 minutes while the reboot completes. Update 8:56 AM: Service has been restored 100% on Scrooge
We received reports from partners on the DEWEY Exchange 2007 about performance issues. We’ve scheduled a reboot to happen at Midnight (6/10/2010). Service is only expected to be impacted for 15 minutes while the reboot completes.
Synchronization between blackberry devices and the LOUIE server is currently in the ‘critical-interrupted’ state, as new mail is not being delivered to devices. We are currently working with Microsoft and RIM to pin point the issue, but it appears that our quickest option is to reinstall the BES server. We do not anticipate the need for BB devices to be re-parried with the BES server is a reinstallation is necessary. Update 9:00 PM Eastern: Service has been restored on the LOUIE BES server. Partners on the LOUIE BES server should have received emails from OWN Support containing the new activation passwords for users.
Earlier this morning we had a few reports that users were not able to send outgoing emails from their BB devices. We located and resolved the issue. As an additional measure, we’ll be moving the BES server to our newer Exchange 2010 cluster. Starting at 2 PM Eastern users will be unable to synchronize from their BB devices until the migration is completed. We anticipate the migration only taking 30 minutes. Update 2:08 PM Eastern: The server is about to go offline for the migration. Update 3:10 PM Eastern: The migration is taking longer than anticipated, however, leaving it on the current host is only providing a consistent experience for a majority of users, and not all; we have to finish the migration. Update 6:17 PM Eastern: Service has been restored 100% and contact with all user BB devices has completed.
We’ve had to temporarily pause the Ahsay OBM Service as we migrate a few users between drives. Service is expected to be back to 100% operation by 5pm Eastern.
We are currently in the process of moving one of the LOUIE CAS node servers to a new host. Earlier we had alerts from Exchange monitoring that was pausing the Exchange processes. We expect service to only be temporarily interrupted, and the louie network should be in normal operation in less than 15 minutes. Later on tonight there will be an additional reboot needed on the CAS array nodes to apply changes to our thresholds/monitoring. Update 9:49 PM Eastern: The CAS array is about to go offline for the update of the monitoring software. Service on louie should be back to 100% functionality by 10:30 PM Eastern. Update 10:42 PM Eastern: The updates are completed and the CAS array nodes are coming back online.
We are currently addressing an issue where the webmail site for mail1.ownwebnow.com (Webhosting mail server). We do not anticipate the outage lasting more than 20 minutes. Update 11:23 AM Eastern: We’ve detected a hardware issue on mail1. In the interest of uptime during peek hours for our customers, we’ve decided to hold off on fully fixing the issue until tonight. Throughout the day, customers may experience brief periods of inaccessibility as the server soft-recovers itself.
Earlier this morning we had an outbreak of spam sent through the ExchangeDefender outbound grid. As a result, one of our IPs was listed on an RBL. As a temporary solution, we’ve disabled two of the smart host nodes as we isolate and remove the spam messages. There may be a minor delay in outbound emails, but we anticipate being back to 100% operation by 2:30 PM Eastern.
Tonight, May 17th 2010 at 10 PM Eastern, we will be rebooting the DEWEY server to address issues with the OWA search indexer, and the amount of resources utilized by the Exchange store. Access to mailboxes will be limited and the downtime is proposed to be under 15 minutes. Update 9:50 PM Eastern: We are beginning to stop the Exchange Services in preparation for the reboot. Customers needing immediate access to their mail can utilize LiveArchive during the reboot. Update 10:14 PM Eastern: The reboot is complete and all services are back online
The Exchange 2007 server DEWEY will be rebooting at 11PM EST tonight to resolve an issue with the OWA Search Indexer that is returning no results.
We are in the process of finalizing the upgrades to LiveArchive 3.0. Throughout the week we received complaints from a few partners about delays in mail being delivered to the server and mail being delivered to livearchive. We identified the problem as a bottleneck, which we implemented a temporary solution of la-edge. During this final upgrade we will have to reboot servers in the livearchive network once before installing the new transport server. Throughout the night we will be flushing out overflow queues from the past 48 hour into LiveArchive mailboxes.
On Saturday, April 17th at 9 PM EST We will being the process of moving one of the mailbox databases to a upgraded RAID array. During this time, users on DEWEYMBOX1 will be unable to connect to their mailboxes through Outlook or OWA. We strongly urge partners to familiarize their clients that require constant access to their email with LiveArchive as this will contain live mail during the downtime. Update 8:27 PM: The database move is confirmed to begin at 9PM EST. During the move, users on MBOX1 will be unable to access their mailboxes. Update 12:14 AM EST: The database move has completed. All service has returned to 100% operation.
To address several new SPAM patterns and strains that are able to get around the current restrictions (and whitelist implementations) we will be rolling out an update to our AntiSPAM engine at 11:00 EST tonight. During the 15 minute window around the switch, there may be an unusual amount of SPAM passing through as our databases are flushed, reloaded and propagated through the network.
This Sunday, 4-11-10 at 7PM EST We will be dismounting MailboxDB3 on the DEWEY exchange server to move it back to the primary RAID array. During this period, users on MailboxDB3 will be unable to access their mailboxes, however, users needing immediate access to their mailbox can utilize LiveArchive to retrieve their mail. |
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
