HUEY and DEWEY are currently offline. While working with Microsoft to troubleshoot an authentication issue, the servers went lights out. We’ve requested a reboot at the DC an service is expected to be restored within 15 minutes. Update 4:57 PM: Service has been restored. Update 4:36 PM: The server took a bit longer than expected to recover from the reboot, however the information store timed out when mounting. We are rebooting the server again to address the issue.
We’ve received complaints from users about inaccessibility to access backup74. We will be rebooting backup74 shortly to resolve the issue.
We will be rebooting two of our US Exchange servers at 8:00 PM Eastern. HUEY and DEWEY will be unavailable for 15 minutes during the reboot. Update 8:20 PM: Service on Dewey has been restored. Update 8:13 PM: Huey’s services have been restored. Dewey hasn’t rebooted yet. Update 8:05 PM: The reboot process has begun. Service will be restored momentarily.
Our Australian Exchange 2007 server is currently down hard. The datacenter is experiencing a networking issue and we are working to resolve the issue. Update 6:15 PM Eastern: The server is back online. The issue was caused by the network ISP change at the data center and the downtime was unexpected. We will be changing the IP address of Australia to match the new network configuration. This change will not require any modification by our partners.
Our shared POP3 mailbox server mail1 is currently under an extreme load and slowly processing messages. We are working to alleviate the queue load and restore mail routing. Update 1:11 PM: The load on the mail1 server has been reduced and new mail is being processed successfully. We anticipate issues for some users connecting as all users are concurrently connecting to download mail.
We will be rebooting our US Exchange Server Scrooge today to relieve resource usage on the server. The reboot time has not been established yet however this post will be update when a time has been set. Update 12:25 AM: Scrooge has been rebooted and is back online. Update 5:22 PM: We have delayed the scheduled reboot until Midnight tonight due to concerns of unavailability by our West coast customers. Update 2:15 PM: We have delayed the scheduled reboot until 9pm tonight in order to run extra maintenance on the server. Update 10:50 AM: We have scheduled the reboot time to occur at 5pm Eastern today.
Backup74 will be temporarily offline from 11:40am – 1:00 pm as we move accounts to new volumes. This is estimated to take an hour. Update 2:30 PM Eastern: Service has been restored.
Over the weekend our replication servers experienced a larger than normal lag in replicating statistical data from our Dallas Colo. This caused quarantine reports to show up as empty as there was no new data replicated when the reports were generated. Reporting is expected to be fully functional by the next scheduled cycle.
The Australian exchange server is currently unavailable. There is an issue with the datacenter and it has gone beyond the normal 1 hour SLA. We are doing everything in our power to restore service immediately. Update 1:34 PM Eastern: The Australian server is back online. The issue was caused by a failure in the core network of the datacenter. Service is now 100% operational.
On Friday, September 4th 2009 at 10:00 AM Eastern we will be changing the IP address for our Australia Exchange server due to changes in the network layout in the data center. Before the change we will lower the TTL time on the DNS record (exchange.australia.exchangedefender.com) to minimize the chance of downtime/unavailability. Users on the Australia server should not have to make any changes after the IP address change for service to continue working. Update 12:10 PM Eastern: The IP change has been delayed until later today. We will update this post before the change occurs.
backup74 will be temporarily unavailable Tuesday Aug 25th from 1pm-4pm Eastern as we apply a new hotfix to solve reporting issues. Update 5:31 PM Eastern: Maintenance has completed and the server is online. Update 4:21 PM Eastern: Maintenance is running a bit over schedule, however it’s close to being completed. Service is expected to be fully functional by 5pm. Update 10:43 AM Eastern: Our maintaince schedule will be beginning earlier than expected to ensure our schedule is completed before peak time. The server will be unavailable until 4 PM Eastern today.
We are rebooting launchpad.exchangedefender.com, our US sharepoint 2007 server. This reboot is due to a failed installation of an sharepoint update.
We have been running an extended maintenance interval process on ExchangeDefender to run some optimization processes with regard to PSA integration. Some users have reported that they were unable to access admin.exchangedefender.com, something we have no explanation for at the moment as we’ve not had any service interruption on it (just high loads and periodic latency). Our Exchange 2007 network is also under extended maintenance due to the issues we’ve experienced between Exchange 2007 Update Rollup 9 and Blackberry Enterprise Services. We are still working on it and will provide updates as we have them.
We will be performing a power (electricity) upgrade during the regularly scheduled monthly interval tomorrow at 11 AM EST – Noon EST (Saturday, July 24). This upgrade will affect a portion of our ExchangeDefender hosting network, including Exchange hosting service on HUEY network. While we do not expect an unanticipated service outage, we will be testing the remote reboot switch and service recovery of the new power system including a lights-out test (full power outage, followed by a power restore). We are conducting these tests and maintenance intervals to make sure we never interrupt you during work hours and keep our streak of rock solid Exchange hosting performance. It comes with a price tag of expensive hardware and extensive maintenance. Thank you for your understanding and your business.
Backup80 (Europe) service will be temporarily interrupted as we install the latest service pack release from Ahsay. Update 12:41 PM Eastern: The upgrade is complete. Ahsay OBS is now running at 5.5.3.0
We’ve received complaints from multiple partners that ExchangeDefender reports were not sent at the previously scheduled time. We are investigating the cause and working towards a resolution.
The livearchive server will be going down for a reboot momentarily as we resolve queue related issues. Updated: 10:47 PM Eastern: After multiple attempts to quickly recover the database, the store still has performance issues mounting. We are going to work out a plan with Microsoft to repair the database and then merge it with the running (new) mailstore. We do not consider any data to be lost. We have 6 TB of a mailstore to repair. We estimate that it could take up to two weeks to merge the data together. Updated 4:09 PM Eastern: We’ve restored mail flow and login ability to the server, however, the old mail store is dismounted and being repaired. We plan on merging the current mailstore that is actively running and the old mail store. Updated 2:43 PM Eastern: We have been working with Microsoft for the past hour to resolve the mail delivery and login issue to LiveArchive. There is no current ETA on resolution time.
We anticipate receiving reports about delays in ExchangeDefender for mail accepted in the earlier half of today. In short, the livearchive server experienced a larger than normal queue load, which delayed messages from the inbound grid being copied to livearchive. This issue has been resolved and any queued mail should be delivered momentarily.
Service on backup74 will be temporarly suspended as we apply the newest hotfix from Ahsay which provides support for Windows Server 2008. Update 12:00 PM Eastern: OBS service has been restored and the server is now patched to 5.5.3. To use ahsay with Windows Server 2008, please download the latest OBM agent @ http://www.ahsay.com Update 11:45 AM Eastern: OBS service has been stopped to install the hotfix.
There is an unexpected service outage with the sharepoint server. We are working to restore service. Update 6:50 PM Eastern: The sharepoint server is back online and accepting requests, however, there will be continuing maintenance on the server as we finish resolving the issue. Update 3:30PM Eastern: We are working with Microsoft troubleshoot the issue. Update 1:38PM Eastern: We have opened a urgent case with Microsoft to resolve this issue. |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
