Tonight we will be completing the migration efforts from last week. At 9 PM Eastern we will be dismounting the mailbox databases and move them to the new RAID array on the new server. The move is expected to take about 3 hours to complete.

Update 9:30 PM Eastern: We are beginning the database move on Scrooge. Service is expected to be restored by 12:30 AM Eastern.

Update 1:40 AM Eastern: The database move is still in progress. Currently there is no ETA on a restore of service, however service will be restored before the start of business.

Last Friday we installed a new HUB server to the DEWEY network to alleviate pressure noticed by users. Unfortunately, the quickest and largest yielding result we could perform was to kill the transport services on DEWEY’s main hub, and offload it to a SMTP array of servers.

Because of the changes required, users can no longer submit messages via SMTP to 65.99.255.12 and must change their outgoing SMTP server to hub.dewey.exchangedefender.com

This change should only affect users connecting to DEWEY through non MAPI or non RPC/HTTPs connections, such as Mac clients, IMAP4 and POP3 users.

We received a couple reports of inhibited mail flow on the DEWEY server, with some clients reporting that they were disconnected from the server. Upon investigation, the transport service had a poisoned message in queue that locked up mail flow. After removing the message, clients reported they were able to connect.

The mail flow issues occurred between 5:30 PM Eastern and 6:00 PM Eastern

On Sunday August 22nd we will be migrating the HUEY and SCOOGE Exchange 2007 Servers to new Dell servers. During the upgrade clients will not be able to access their mailboxes via outlook, active sync or OWA. Customers needing constant access to their mailboxes should become familiar with using LiveArchive before the outage occurs.

Maintenance is scheduled to begin at 5PM Eastern and is likely to continue until the early AM on Monday. Service will be restored prior to 9AM on Monday August 23rd.

Update 12:35 AM Eastern Monday: We are beginning the upgrade process on HUEY tonight. Access will be unavailable to users on the HUEY server during the upgrades.

Update 7:54 AM Eastern Monday: The upgrade process took significantly longer than expected and was terminated to meet the 9AM deadline. The installation will continue today at 830pm Eastern

The DEWEY Exchange 2007 server was reported to have delays in processing requests from clients. We are restarting the information store to resolve the issue.

Update 1:00 PM: The information store has finished restarting and we’ve confirmed access to mailboxes on each DB.

Update 3:46 PM: Multiple parties have reported issues in delivery speed. We are rebooting the DEWEY server in order to clear out the cached memory.

Update 4:08 PM: We’ve been monitoring the server reboot via KVM. The server is in the process of installing windows updates before it completes the reboot.

Update 4:21 PM: The installation of the windows update has completed and the server is now stopping the Exchange services. The server should complete the reboot in the next 15 minutes.

Update 4:49 PM: The reboot has completed. We’ve verified that mail is being routed and confirmed that our test users could login.

Update 5:30 PM: After the reboot, the DEWEY transport started routing mail through one of it’s alias IPs instead of the primary IP, causing outgoing messages to be rejected as “Relaying denied, proper authentication required” The issue has been resolved and mail is routing out.

The outage looks like an LDAP query was null routing, but not dying and eating up resources — We killed the query and was able to establish connection to our test account. Users should be able to reconnect in the next few minutes as the databases reconnect. The outage looks to have started around 11:10 AM and was resolved by 11:20 AM.

Our Exchange 2007 network DEWEY is currently registering as offline in our monitoring software. We are in the process of investigating this issue now.

Update 2:09 PM Eastern: The issue is related to one of the core networks used by DEWEY. We detected and thwarted a DDoS attack. Service is back to 100% operation.

On Saturday July 30st and Sunday Aug 1st we will be performing an all day storage upgrade on the LOUIE exchange mailbox servers. The scheduled updates will provide clients improved speeds in performing API calls to the server, such as sending messages or resolving the GAL.

During the upgrade clients will experience disconnections from their mailbox as their database moves to the new storage medium.

Clients requiring on-demand access to their mail should utilize LiveArchive (https://livearchive.exchangedefender.com) if they are unable to access their mailbox on the LOUIE network.

Update Saturday 3:30 PM Eastern: We are about to begin the hardware installation for the LOUIEMBOX servers. During the installation, customers will be unable to reach the Exchange server. We expect the downtime for the upgrade to be no more than 20 minutes.

Update Saturday 4:30 PM Eastern: The hardware installation completed about 30 minutes ago and everything is going as expected. Throughout the night we will be preparing the new storage array before the databases are moved.

Update Sunday 1:10 PM Eastern: The storage array initialization has completed. We are beginning the process of moving mailbox databases to the new storage array. During the mailbox move, customers may experience disconnection from Outlook and OWA, however, the migration is estimated to take up to an hour and a half per database.

Update Sunday 1:19 PM Eastern: The first mailbox database has finished copying to the new storage array. The nest database will take about two hours to complete moving.

Update Sunday 4:44 PM Eastern: After successfully moving the databases, we discovered that the backup software cannot work with our drive structure. We are moving the databases to a temporary holding spot as we restructure the drives.

Update Sunday10:05 PM Eastern: The upgrades to LOUIE have been complete and everything was migrated successfully. We appreciate our partners and their customer’s patience as this upgrade went in place.

We are conducting an emergency reboot on the LOUIE GC servers, which includes one of the mailbox servers and a CAS array because of an inconsistency in the GC. Customers will lose connection from Outlook for up to 10 minutes, but will reconnect after the reboot is complete.

Update 1:54 PM Eastern: The servers are back from the reboot, but service has not been restored. We do not estimate the outage to last more than 10 more minutes.

Update 2:00 PM: Service has been restored on the LOUIE network. We are making a final reboot on the Blackberry server for LOUIE.

The Exchange 2007 DEWEY server is currently experiencing issues with IIS. We are working to restore connectivity and do not expect the outage to last longer than an hour. Clients needing immediate access to email can utilize livearchive @ https://livearchive.exchangedefender.com/owa

Update 5:32 PM Eastern: Service has been restored to DEWEY

Throughout the night we will be installing new drives to a few LOUIE servers. The following services will be intermittently interrupted throughout the upgrade which is scheduled to last between 5:30 PM and 7:00 PM

Exchange:

The CAS array host will be rebooted which will leave Outlook disconnected for up to five minutes

BES:

The BES database is being moved to a new drive which will leave BB devices unable to synchronize for up to 30 minutes

Sharepoint:

The Sharepoint SQL host is being moved to a new drive which will leave Sharepoint portals unavailable for up to five minutes

Update 7:14 PM Eastern: The database move for the BES server is taking longer than expected. We are still working to finalize the upgrades as soon as possible.

Update 8:02 PM Eastern: The CAS array is being upgraded now. Users will experience two brief disconnects which are scheduled to last up to 10 minutes each.

At 9:00 PM Eastern tonight (6/22/2010) we will be installing the long awaited Exchange 2010 Rollup 4 patch which brings back the ability for PST imports through automation. Service is expected to be lightly impacted between 9:00 PM – 9:30 PM as we patch all seven servers.

Update 6/23/10: The patch rollup installation was delayed last night and moved to today. We will begin the patch process at the same time, 9:00 PM Eastern.

Update 10:47 PM: The final patch has been installed and the final reboot is commencing. Service is expected to be 100% operational by 11 PM Eastern.

SCROOGE (Exchange 2007) will be going down momentarily for a reboot after tripping our monitoring threshold a few times. The reboot is being done as a proactive measure to avoid any issues during ‘business hours’.

The server is expected to only be down for 15 minutes while the reboot completes.

Update 8:56 AM: Service has been restored 100% on Scrooge

We received reports from partners on the DEWEY Exchange 2007 about performance issues. We’ve scheduled a reboot to happen at Midnight (6/10/2010). Service is only expected to be impacted for 15 minutes while the reboot completes.

Synchronization between blackberry devices and the LOUIE server is currently in the ‘critical-interrupted’ state, as new mail is not being delivered to devices. We are currently working with Microsoft and RIM to pin point the issue, but it appears that our quickest option is to reinstall the BES server. We do not anticipate the need for BB devices to be re-parried with the BES server is a reinstallation is necessary.

Update 9:00 PM Eastern: Service has been restored on the LOUIE BES server. Partners on the LOUIE BES server should have received emails from OWN Support containing the new activation passwords for users.

Earlier this morning we had a few reports that users were not able to send outgoing emails from their BB devices. We located and resolved the issue. As an additional measure, we’ll be moving the BES server to our newer Exchange 2010 cluster. Starting at 2 PM Eastern users will be unable to synchronize from their BB devices until the migration is completed. We anticipate the migration only taking 30 minutes.

Update 2:08 PM Eastern: The server is about to go offline for the migration.

Update 3:10 PM Eastern: The migration is taking longer than anticipated, however, leaving it on the current host is only providing a consistent experience for a majority of users, and not all; we have to finish the migration.

Update 6:17 PM Eastern: Service has been restored 100% and contact with all user BB devices has completed.

We are currently in the process of moving one of the LOUIE CAS node servers to a new host. Earlier we had alerts from Exchange monitoring that was pausing the Exchange processes. We expect service to only be temporarily interrupted, and the louie network should be in normal operation in less than 15 minutes. Later on tonight there will be an additional reboot needed on the CAS array nodes to apply changes to our thresholds/monitoring.

Update 9:49 PM Eastern: The CAS array is about to go offline for the update of the monitoring software. Service on louie should be back to 100% functionality by 10:30 PM Eastern.

Update 10:42 PM Eastern: The updates are completed and the CAS array nodes are coming back online.

Tonight, May 17th 2010 at 10 PM Eastern, we will be rebooting the DEWEY server to address issues with the OWA search indexer, and the amount of resources utilized by the Exchange store. Access to mailboxes will be limited and the downtime is proposed to be under 15 minutes.

Update 9:50 PM Eastern: We are beginning to stop the Exchange Services in preparation for the reboot. Customers needing immediate access to their mail can utilize LiveArchive during the reboot.

Update 10:14 PM Eastern: The reboot is complete and all services are back online

The Exchange 2007 server DEWEY will be rebooting at 11PM EST tonight to resolve an issue with the OWA Search Indexer that is returning no results.

On Saturday, April 17th at 9 PM EST We will being the process of moving one of the mailbox databases to a upgraded RAID array. During this time, users on DEWEYMBOX1 will be unable to connect to their mailboxes through Outlook or OWA. We strongly urge partners to familiarize their clients that require constant access to their email with LiveArchive as this will contain live mail during the downtime.

Update 8:27 PM: The database move is confirmed to begin at 9PM EST. During the move, users on MBOX1 will be unable to access their mailboxes.

Update 12:14 AM EST: The database move has completed. All service has returned to 100% operation.