Tonight we will be completing the migration efforts from last week. At 9 PM Eastern we will be dismounting the mailbox databases and move them to the new RAID array on the new server. The move is expected to take about 3 hours to complete. Update 9:30 PM Eastern: We are beginning the database move on Scrooge. Service is expected to be restored by 12:30 AM Eastern. Update 1:40 AM Eastern: The database move is still in progress. Currently there is no ETA on a restore of service, however service will be restored before the start of business.
Last Friday we installed a new HUB server to the DEWEY network to alleviate pressure noticed by users. Unfortunately, the quickest and largest yielding result we could perform was to kill the transport services on DEWEY’s main hub, and offload it to a SMTP array of servers. Because of the changes required, users can no longer submit messages via SMTP to 65.99.255.12 and must change their outgoing SMTP server to hub.dewey.exchangedefender.com This change should only affect users connecting to DEWEY through non MAPI or non RPC/HTTPs connections, such as Mac clients, IMAP4 and POP3 users.
Tonight starting at 8:30 PM Eastern we will begin upgrading the DEWEY Exchange 2007 network by installing additional domain controllers and additional hub transport servers. During the upgrades, we will have to reboot the mailbox and primary DC for DEWEY. Users may experience brief periods of disconnection from the Exchange server during the upgrades. Update 8:35 PM Eastern: We are about to reboot the primary mailbox server for DEWEY. Connectivity is expected to be degraded for up to 20 minutes.
We are beginning the process of continuing the upgrades to HUEY and Scrooge. We are beginning again with HUEY and will move on to Scrooge. Just as yesterday, access to mailboxes through Exchange will be interrupted until the upgrade is complete. Each server is scheduled to take about 4 hours. Ref: http://www.ownwebnow.com/noc/2010/08/23/huey-and-scrooge-upgrades/ Update 1:37 AM Eastern: The imaging of the OS on HUEY was successful. We are in the process of moving the database and beginning work on Scrooge Update 3:18 AM Eastern: The DB copy on HUEY is still in progress and is about half way completed. Since the information is being written to a new RAID volume, the copy process must write to all disks in the array. Scrooge has about 35 minutes on the OS image. Update 4:42 AM Eastern: The DB copy has begun on HUEY and windows estimates 2.5 hours for completion. We are about to begin the image of Scrooge Update 5:00 AM Eastern: The DB copy completed on HUEY and service has been restored. Scrooge is committing the backup image to the new server and should be ready for the database copy in 30 minutes. Update 7:44 AM Eastern: All upgrades were a stunning success. The upgrade to HUEY is complete and the databases were moved. Scrooge is on the new hardware, however the databases copy was taking longer than our allowed threshold. We’ve moved the old database drives to the new server and will complete the DB move either tonight or over the weekend. Service is expected to be operational by 8AM Eastern, but may be slightly reduced until 8:30 as both servers process pending mail. Update 8:08 AM Eastern: As a precautionary action we are running a chkdsk on the Scrooge DB drives to ensure they will be able to serve clients without issue. The scan is already 30 minutes in and should be completing by 9AM. Update 8:55 AM Eastern: We are going to let the chkdsk run until 9:30 AM Eastern before we look into switching to the backup drive. We appreciate our partners patience as this upgrade will provide clients with a smoother Exchange experience. Update 9:25 AM Eastern: Running a quick reboot in HUEY to correct the system time clock Update 9:32 AM Eastern: The reboot on HUEY completed and the information store is coming online. If any clients experience mail “disappearing” after it shows up as new, please check the mail from Yesterday in Outlook as the time on the server was one day behind. Update 10:00 AM Eastern: The chkdsk on the drive is on the final stage, currently at 30%. The server should be 100% operational by 10:30 AM. Canceling the chkdsk was not optional as the other drive presented the same issue as the original. Update 10:22 AM Eastern: The chkdsk is currently at 50%. This changes the final time to a more realistic 11AM. Customers needing instant access to their mail should utilize LiveArchive until the upgrades are complete. The upgrades were not scheduled to last past 9AM, however we needed to complete the physical migration today. Update 10:58 AM Eastern: chkdsk is currently at 80% and looks like the slow down was due to a couple logical corrupted free-space allocation blocks. Update 11:50 AM Eastern: The physical upgrade is complete and service is 100% operational.
We received a couple reports of inhibited mail flow on the DEWEY server, with some clients reporting that they were disconnected from the server. Upon investigation, the transport service had a poisoned message in queue that locked up mail flow. After removing the message, clients reported they were able to connect. The mail flow issues occurred between 5:30 PM Eastern and 6:00 PM Eastern
On Sunday August 22nd we will be migrating the HUEY and SCOOGE Exchange 2007 Servers to new Dell servers. During the upgrade clients will not be able to access their mailboxes via outlook, active sync or OWA. Customers needing constant access to their mailboxes should become familiar with using LiveArchive before the outage occurs. Maintenance is scheduled to begin at 5PM Eastern and is likely to continue until the early AM on Monday. Service will be restored prior to 9AM on Monday August 23rd. Update 12:35 AM Eastern Monday: We are beginning the upgrade process on HUEY tonight. Access will be unavailable to users on the HUEY server during the upgrades. Update 7:54 AM Eastern Monday: The upgrade process took significantly longer than expected and was terminated to meet the 9AM deadline. The installation will continue today at 830pm Eastern
The DEWEY Exchange 2007 server was reported to have delays in processing requests from clients. We are restarting the information store to resolve the issue. Update 1:00 PM: The information store has finished restarting and we’ve confirmed access to mailboxes on each DB. Update 3:46 PM: Multiple parties have reported issues in delivery speed. We are rebooting the DEWEY server in order to clear out the cached memory. Update 4:08 PM: We’ve been monitoring the server reboot via KVM. The server is in the process of installing windows updates before it completes the reboot. Update 4:21 PM: The installation of the windows update has completed and the server is now stopping the Exchange services. The server should complete the reboot in the next 15 minutes. Update 4:49 PM: The reboot has completed. We’ve verified that mail is being routed and confirmed that our test users could login. Update 5:30 PM: After the reboot, the DEWEY transport started routing mail through one of it’s alias IPs instead of the primary IP, causing outgoing messages to be rejected as “Relaying denied, proper authentication required” The issue has been resolved and mail is routing out.
On Sunday August 22nd we will be migrating the HUEY and SCOOGE Exchange 2007 Servers to new Dell servers. During the upgrade clients will not be able to access their mailboxes via outlook, active sync or OWA. Customers needing constant access to their mailboxes should become familiar with using LiveArchive before the outage occurs. Maintenance is scheduled to begin at 5PM Eastern and is likely to continue until the early AM on Monday. Service will be restored prior to 9AM on Monday August 23rd. Update 12:35 AM Eastern Monday: We are beginning the upgrade process on HUEY tonight. Access will be unavailable to users on the HUEY server during the upgrades.
Around 4:02 PM Eastern we started receiving alerts from our monitoring system about packet loss on the network. Upon investigation, we found an IP that was flooding with fragmented packets..the ICS disabled the talker, but the backlog of traffic caused some unavailability to clients.
The outage looks like an LDAP query was null routing, but not dying and eating up resources — We killed the query and was able to establish connection to our test account. Users should be able to reconnect in the next few minutes as the databases reconnect. The outage looks to have started around 11:10 AM and was resolved by 11:20 AM.
Access to livearchive is currently deprecated. Yesterday we noticed mail submission would randomly drop, so we increased our monitoring. Today, we’re noticing the same behavior, but it’s pointing more to the mailbox databases. We are currently taking multiple databases off at a time and running integrity checks. We highly apologize for the inconvenience and any partner who has a down server that needs access to livearchive, please open a case with support and we will work out an alternative solution.
We received alerts from our monitoring software that messages on outbound1 were being rejected as relaying denied. We killed the sendmail process and resolved the issue. The bounce backs occurred between 11:20 AM Eastern and 11:35 AM Eastern. Update 6:44 PM Eastern: The issue reappeared around 6:20 PM and began rejecting messages. We located the root cause, which was a faulty update from an AV vendor. We applied the previous definitions and await an update from the vendor.
Access to livearchive will be temporarily unavailable for some users as we dismount databases to make schema level changes for our new monitoring. Service is expected to be fully operational by noon Eastern. Update 12:00 PM Eastern: Service to livearchive is back to normal operation Update 1:54 PM Eastern: We’re receiving reports that some clients cannot access livearchive. We are actively investigating the issue. Update 2:48 PM Eastern: As part of our investigation, we have to disable mail routing to livearchive. Customers should expect to see a gap in mail in their livearchive mailboxes. Update 3:09 PM Eastern: We are installing more memory modules into the primary mbox server for livearchive. There is no ETA as of yet.
Our Exchange 2007 network DEWEY is currently registering as offline in our monitoring software. We are in the process of investigating this issue now. Update 2:09 PM Eastern: The issue is related to one of the core networks used by DEWEY. We detected and thwarted a DDoS attack. Service is back to 100% operation.
We are in the process of installing Exchange 2007 SP3 to the EUROPE and DEWEY server. Previous attempts to install the SP failed until a resolution was discovered earlier today. Clients may experience a brief disconnection from outlook, but service will resume in less than 20 minutes.
On Saturday July 30st and Sunday Aug 1st we will be performing an all day storage upgrade on the LOUIE exchange mailbox servers. The scheduled updates will provide clients improved speeds in performing API calls to the server, such as sending messages or resolving the GAL. During the upgrade clients will experience disconnections from their mailbox as their database moves to the new storage medium. Clients requiring on-demand access to their mail should utilize LiveArchive (https://livearchive.exchangedefender.com) if they are unable to access their mailbox on the LOUIE network. Update Saturday 3:30 PM Eastern: We are about to begin the hardware installation for the LOUIEMBOX servers. During the installation, customers will be unable to reach the Exchange server. We expect the downtime for the upgrade to be no more than 20 minutes. Update Saturday 4:30 PM Eastern: The hardware installation completed about 30 minutes ago and everything is going as expected. Throughout the night we will be preparing the new storage array before the databases are moved. Update Sunday 1:10 PM Eastern: The storage array initialization has completed. We are beginning the process of moving mailbox databases to the new storage array. During the mailbox move, customers may experience disconnection from Outlook and OWA, however, the migration is estimated to take up to an hour and a half per database. Update Sunday 1:19 PM Eastern: The first mailbox database has finished copying to the new storage array. The nest database will take about two hours to complete moving. Update Sunday 4:44 PM Eastern: After successfully moving the databases, we discovered that the backup software cannot work with our drive structure. We are moving the databases to a temporary holding spot as we restructure the drives. Update Sunday10:05 PM Eastern: The upgrades to LOUIE have been complete and everything was migrated successfully. We appreciate our partners and their customer’s patience as this upgrade went in place.
We are conducting an emergency reboot on the LOUIE GC servers, which includes one of the mailbox servers and a CAS array because of an inconsistency in the GC. Customers will lose connection from Outlook for up to 10 minutes, but will reconnect after the reboot is complete. Update 1:54 PM Eastern: The servers are back from the reboot, but service has not been restored. We do not estimate the outage to last more than 10 more minutes. Update 2:00 PM: Service has been restored on the LOUIE network. We are making a final reboot on the Blackberry server for LOUIE.
Throughout the night we will be performing upgrades to the backup system for the 2010 Exchange LOUIE network. Users may experience brief delays when interacting with their mailbox, but no mail will be lost during this upgrade. If a user becomes disconnected from outlook, advise them to either use OWA (https://cas.louie.exchangedefender.com/owa) or LiveArchive (https://livearchive.exchangedefender.com) The projected window is from 11pm EST – 3am EST. We will upgrade this post in the event of any scheduled outages.
Two days ago our first sharepoint 2007 server launchpad (serves Scrooge, HUEY, Europe and Australia) had a failed drive in the RAID array. The replacement drive arrived today, and we will be taking the server offline at 9:00 PM Eastern to rebuild the RAID array. The rebuild is scheduled to take up to two hours, which will leave sharepoint services on launchpad unavailable.
The Exchange 2007 DEWEY server is currently experiencing issues with IIS. We are working to restore connectivity and do not expect the outage to last longer than an hour. Clients needing immediate access to email can utilize livearchive @ https://livearchive.exchangedefender.com/owa Update 5:32 PM Eastern: Service has been restored to DEWEY |
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
