Update 4:00 AM Eastern: The database copy is still underway. At this point we are planning on letting the DB copy finish throughout the Friday day as we cannot afford to let the DB copy extend past this weekend.

Users on Scrooge will see Empty mailboxes throughout the day, however rest assured that your previous mailbox is still intact. Over the weekend we will converge all user mailboxes.

Clients using Outlook should have their cached profiles throughout the day that will enable them to work like normal. Clients using OWA are recommended to concurrently use LiveArchive to access mail prior to the DB move.

Update 5:10 AM Eastern: We’ve decided not to reset user mailboxes, and instead run of the current drive for one more day. We understand that users will continue to experience delays, however we feel it would be in the best interest of all users to complete the DB move over the weekend instead of doing a converge.

Update 5:28 AM Eastern: We’ve re-enabled the previous DBs. For the time being, public folders will be unavailable on the server to try and save as much reads/writes as possible on the server until the backup restarts tonight.

Friday 9/3/10

Update 11:42 PM Eastern: We are about to restart the database migration. We anticipate the migration taking at least 12 hours. Customers should use LiveArchive if they need access to their live mail.

Sunday 9/5/10

Update 1:25 PM Eastern: The database backup was successfully completed and we began the process of restoring the backup to the new RAID array. We anticipate mailboxes being available by 7PM Eastern tonight and public folders remounted tomorrow afternoon.

Update 5:45 PM Eastern: We’ve had a few inquires from partners on the lack of updates with the work done yesterday and today. While we love keeping this NOC blog updated with fairly live updates, the majority of time in the past two days has been watching databases and users copy between drives. We do appreciate everyones patience as we attempt to finalize all changes by the end of the holiday weekend.

Update 6:57 PM Eastern: The second attempt at the restore is still in progress. The earlier restore slowed down about half way through, so we restarted the server and disconnected the older RAID array. The current restore is looking good and is estimated to complete in 45 minutes. After the restore is complete, we are going to try to mount the database and then move the databases to it’s final location.

Update 7:50 PM Eastern: The restore was completed and we’ve enabled OWA access to Scrooge. All pending mail is being delivered to the user mailboxes and Outlook access will be enabled after the queues are flushed out. We’re going to test a DB move to the array at 10 PM, and  once an estimated time is established we will update this post.

Update 9:27 PM Eastern: Service has been restored and we’ve postponed the test move and we will update this post.

We will be taking requests to move clients via support requests starting Tuesday 10/06/10

Clients should not have to reconfigure outlook as long as an autodiscover record was created.

During a mailbox move, the mailbox will be unavailable to Outlook or OWA until the move completes. It generally takes about an hour to process two gigs, but can be lower or higher depending on the item count and the mailbox structure.

Friday 9/3/10

Update 11:45 PM Eastern: We are beginning to move users to the new database server.

Update 9:10 PM Eastern: The installs went smoothly and service is 100% operational.

Update 3:30 PM: The server has been online for the past twenty minutes, but we noticed an extremely large amount of traffic going from DEWEY. Upon investigation, it looks like DEWEY was encountering issues delivering RPC packets to a client who has a misconfigured router as the packets are going back to DEWEY and creating an infinite loop. We’ve disabled traffic to that host and will continue to monitor this over the next 24 hours.

Update 9:30 PM Eastern: We are beginning the database move on Scrooge. Service is expected to be restored by 12:30 AM Eastern.

Update 1:40 AM Eastern: The database move is still in progress. Currently there is no ETA on a restore of service, however service will be restored before the start of business.

Because of the changes required, users can no longer submit messages via SMTP to 65.99.255.12 and must change their outgoing SMTP server to hub.dewey.exchangedefender.com

This change should only affect users connecting to DEWEY through non MAPI or non RPC/HTTPs connections, such as Mac clients, IMAP4 and POP3 users.

Update 8:35 PM Eastern: We are about to reboot the primary mailbox server for DEWEY. Connectivity is expected to be degraded for up to 20 minutes.

Ref: http://www.ownwebnow.com/noc/2010/08/23/huey-and-scrooge-upgrades/

Update 1:37 AM Eastern: The imaging of the OS on HUEY was successful. We are in the process of moving the database and beginning work on Scrooge

Update 3:18 AM Eastern: The DB copy on HUEY is still in progress and is about half way completed. Since the information is being written to a new RAID volume, the copy process must write to all disks in the array. Scrooge has about 35 minutes on the OS image.

Update 4:42 AM Eastern: The DB copy has begun on HUEY and windows estimates 2.5 hours for completion. We are about to begin the image of Scrooge

Update 5:00 AM Eastern: The DB copy completed on HUEY and service has been restored.  Scrooge is committing the backup image to the new server and should be ready for the database copy in 30 minutes.

Update 7:44 AM Eastern: All upgrades were a stunning success. The upgrade to HUEY is complete and the databases were moved. Scrooge is on the new hardware, however the databases copy was taking longer than our allowed threshold. We’ve moved the old database drives to the new server and will complete the DB move either tonight or over the weekend.

Service is expected to be operational by 8AM Eastern, but may be slightly reduced until 8:30 as both servers process pending mail.

Update 8:08 AM Eastern: As a precautionary action we are running a chkdsk on the Scrooge DB drives to ensure they will be able to serve clients without issue. The scan is already 30 minutes in and should be completing by 9AM.

Update 8:55 AM Eastern: We are going to let the chkdsk run until 9:30 AM Eastern before we look into switching to the backup drive. We appreciate our partners patience as this upgrade will provide clients with a smoother Exchange experience.

Update 9:25 AM Eastern: Running a quick reboot in HUEY to correct the system time clock

Update 9:32 AM Eastern: The reboot on HUEY completed and the information store is coming online. If any clients experience mail “disappearing” after it shows up as new, please check the mail from Yesterday  in Outlook as the time on the server was one day behind.

Update 10:00 AM Eastern: The chkdsk on the drive is on the final stage, currently at 30%. The server should be 100% operational by 10:30 AM. Canceling the chkdsk was not optional as the other drive presented the same issue as the original.

Update 10:22 AM Eastern: The chkdsk is currently at 50%. This changes the final time to a more realistic 11AM. Customers needing instant access to their mail should utilize LiveArchive until the upgrades are complete. The upgrades were not scheduled to last past 9AM, however we needed to complete the physical migration today.

Update 10:58 AM Eastern: chkdsk is currently at 80% and looks like the slow down was due to a couple logical corrupted free-space allocation blocks.

Update 11:50 AM Eastern: The physical upgrade is complete and service is 100% operational.

The mail flow issues occurred between 5:30 PM Eastern and 6:00 PM Eastern

Maintenance is scheduled to begin at 5PM Eastern and is likely to continue until the early AM on Monday. Service will be restored prior to 9AM on Monday August 23rd.

Update 12:35 AM Eastern Monday: We are beginning the upgrade process on HUEY tonight. Access will be unavailable to users on the HUEY server during the upgrades.

Update 7:54 AM Eastern Monday: The upgrade process took significantly longer than expected and was terminated to meet the 9AM deadline. The installation will continue today at 830pm Eastern

Update 1:00 PM: The information store has finished restarting and we’ve confirmed access to mailboxes on each DB.

Update 3:46 PM: Multiple parties have reported issues in delivery speed. We are rebooting the DEWEY server in order to clear out the cached memory.

Update 4:08 PM: We’ve been monitoring the server reboot via KVM. The server is in the process of installing windows updates before it completes the reboot.

Update 4:21 PM: The installation of the windows update has completed and the server is now stopping the Exchange services. The server should complete the reboot in the next 15 minutes.

Update 4:49 PM: The reboot has completed. We’ve verified that mail is being routed and confirmed that our test users could login.

Update 5:30 PM: After the reboot, the DEWEY transport started routing mail through one of it’s alias IPs instead of the primary IP, causing outgoing messages to be rejected as “Relaying denied, proper authentication required” The issue has been resolved and mail is routing out.

On Sunday August 22nd we will be migrating the HUEY and SCOOGE Exchange 2007 Servers to new Dell servers. During the upgrade clients will not be able to access their mailboxes via outlook, active sync or OWA. Customers needing constant access to their mailboxes should become familiar with using LiveArchive before the outage occurs.

Maintenance is scheduled to begin at 5PM Eastern and is likely to continue until the early AM on Monday. Service will be restored prior to 9AM on Monday August 23rd.

Update 12:35 AM Eastern Monday: We are beginning the upgrade process on HUEY tonight. Access will be unavailable to users on the HUEY server during the upgrades. ]]> http://www.ownwebnow.com/noc/2010/08/15/huey-and-scrooge-exchange-upgrades/feed/ 0 http://www.ownwebnow.com/noc/2010/08/13/own-network/ http://www.ownwebnow.com/noc/2010/08/13/own-network/#comments Fri, 13 Aug 2010 20:18:04 +0000 travis http://www.ownwebnow.com/noc/2010/08/13/own-network/ Around 4:02 PM Eastern we started receiving alerts from our monitoring system about packet loss on the network. Upon investigation, we found an IP that was flooding with fragmented packets..the ICS disabled the talker, but the backlog of traffic caused some unavailability to clients. 

We highly apologize for the inconvenience and any partner who has a down server that needs access to livearchive, please open a case with support and we will work out an alternative solution.

Update 6:44 PM Eastern: The issue reappeared around 6:20 PM and began rejecting messages. We located the root cause, which was a faulty update from an AV vendor. We applied the previous definitions and await an update from the vendor.

Update 12:00 PM Eastern: Service to livearchive is back to normal operation

Update 1:54 PM Eastern: We’re receiving reports that some clients cannot access livearchive. We are actively investigating the issue.

Update 2:48 PM Eastern: As part of our investigation, we have to disable mail routing to livearchive. Customers should expect to see a gap in mail in their livearchive mailboxes.

Update 3:09 PM Eastern: We are installing more memory modules into the primary mbox server for livearchive. There is no ETA as of yet.

Update 2:09 PM Eastern: The issue is related to one of the core networks used by DEWEY. We detected and thwarted a DDoS attack. Service is back to 100% operation.

During the upgrade clients will experience disconnections from their mailbox as their database moves to the new storage medium.

Clients requiring on-demand access to their mail should utilize LiveArchive (https://livearchive.exchangedefender.com) if they are unable to access their mailbox on the LOUIE network.

Update Saturday 3:30 PM Eastern: We are about to begin the hardware installation for the LOUIEMBOX servers. During the installation, customers will be unable to reach the Exchange server. We expect the downtime for the upgrade to be no more than 20 minutes.

Update Saturday 4:30 PM Eastern: The hardware installation completed about 30 minutes ago and everything is going as expected. Throughout the night we will be preparing the new storage array before the databases are moved.

Update Sunday 1:10 PM Eastern: The storage array initialization has completed. We are beginning the process of moving mailbox databases to the new storage array. During the mailbox move, customers may experience disconnection from Outlook and OWA, however, the migration is estimated to take up to an hour and a half per database.

Update Sunday 1:19 PM Eastern: The first mailbox database has finished copying to the new storage array. The nest database will take about two hours to complete moving.

Update Sunday 4:44 PM Eastern: After successfully moving the databases, we discovered that the backup software cannot work with our drive structure. We are moving the databases to a temporary holding spot as we restructure the drives.

Update Sunday10:05 PM Eastern: The upgrades to LOUIE have been complete and everything was migrated successfully. We appreciate our partners and their customer’s patience as this upgrade went in place.