On April 8th, 2010 @ 9PM EST we will being the process of implementing split GALs on australia.exchangedefender.com During this period users in Outlook may receive warnings about not being able to resolve the name in the address book. This will be a temporary issue shouldn’t last more than 1 hour. Update 2:56 PM EST: We are beginning the process to implement split GALs.
Tonight at 10:00 PM EST DEWEY (Exchange 2007 US Server) will be going undergoing scheduled maintenance, which is estimated to last 4 hours. During the maintenance schedule, DEWEY may be rebooted to update our backup software. Service is expected to not be impacted outside the reboots.
We have posted a remarkable quarter and have focused exclusively on the features and onboarding support. Traditionally, that has been the biggest pain point expressed by our partners and the biggest obstacle to us as a growing organization. In order to provide the level of service and assurance required going forward, we’ve had to reprioritize how we do business in Q4 2009 and Q1 2010. This weekend we will be revealing the first set of patches that have plagued nearly all of our products. Additionally, we are migrating away from several technologies that have, frankly, failed us and failed our partners for the level of quality you should expect from Own Web Now. We are not willing to point any fingers, you and your customers trust us to deliver rock solid solutions and at the end of the day it’s our decision to make the right choices. Unfortunately, the reality is that we haven’t made some right choices in the past two quarters and in April we will be moving to address those issues, in all products across services. There will be extended maintenance hours each weekend outside of business hours as we move to address the many problems noted in the NOC blog over the past few months. As always, I appreciate your business and I speak to partners every day that love what we’re able to do. That is what drives us and my team strives to deliver more every single day. That is what we are good at – competing. We have dedicated 2010 to raising Own Web Now to the next level, so I hope you pardon the dust throughout April as we hold extended maintenance hours to bring a level of consistency. I promise you that each and every step taken, to address some of the issues the few of you have been very vocal about, will be fully communicated through this blog.
We have identified a performance issues on the DEWEY cluster and the team is working on restoring full functionality. While most users are not affected, we recommend using https://livearchive.exchangedefender.com if you are experiencing performance issues. Update: 4:30 PM EST: Performance optimization complete, status for all users is back to optimal. Update: 5:25 PM EST: Users on one of the databases are still reporting issues in delays. We’ve dismounted the mailbox database to check it for inconsistencies. Update 9:13 PM EST: The database checks came back fine and we’ve had the database mounted for a few hours. The server is going down for a reboot to add new hardware and once it’s back online, we will be running a precautionary extra backup of the database. Update 5:16 AM EST: Stats on the server are looking normal and performance is expected to be back to normal operating levels. Throughout the week we will be migrating users across the new sets of databases.
We have already started receiving support requests regarding missing messages – all caused by not including the latest IP addresses in the IP restriction set. If you are currently enforcing restrictions, please make sure you have the 206.125.40.0/24 IP address range in your access list: http://www.ownwebnow.com/blog/2010/03/exchangedefender-grows-again/
Earlier today we launched ExchangeDefender LiveArchive 3, powered by Microsoft Exchange 2010. LiveArchive is a free business continuity solution that is constantly copying all your inbound and outbound messages as they are scanned by ExchangeDefender and in addition to delivering to your mail server or your mail recipients, it delivers to an enterprise deployment of Exchange 2010 in our data centers. When your servers go down, or production environment is interrupted in any way, LiveArchive can be accessed through any browser or mobile web device and users can have access to up to 1 year worth of email (note: not a compliance or archiving solution) Passwords When we went from Exchange 2007 to Exchange 2010 we simply created a new environment and created all the users from ExchangeDefender user databases. You can access your old accounts and your old email at http://livearchive2.exchangedefender.com Your new accounts are at https://livearchive.exchangedefender.com and they all hold the same username as the one in our ExchangeDefender portal. If you changed your passwords via Outlook Web Access keep in mind that the password will now be synced to our ExchangeDefender user databases and only supported passwords are the ones at https://admin.exchangedefender.com. We will soon restrict the ability to change the password through OWA because users forget their passwords and it creates an enormous amount of support issues. If you are running into password issues when logging into LiveArchive today please use your ExchangeDefender password, not your OWA password or anything you may have reset it to via OWA. If you need to change the password please do so at https://admin.exchangedefender.com Have a great weekend and we hope you love Exchange 2010!
We are in the process of cutting over mail flow from our LiveArchive 2 (Exchange 2007) to our LiveArchive 3 (Exchange 2010) cluster. During this transition mail may be delivered to either location, however, users will be able to access their LiveArchive 3 account by 2AM Eastern. Over the course of the next month mail is scheduled to be imported from the previous LiveArchive cluster.
At approximately 1:10 PM EST we started receiving reports of disconnections for Outlook clients. The issue was performance related and although it should not be affecting connectivity at times it does. We restarted the mail store service and that addressed the issue. Update 2:27 PM EST: Some partners are still reporting issues connecting via RPC over HTTPs. We are going to perform a quick reboot on DEWEY to resolve the issue. Update 2:50 PM EST: The issue has been resolved and service has been restored 100%
We are currently attempting to isolate a large scale DDoS on our 65.99.192.0/24 network. This ticket will be updated as soon as we have more information.
We’re currently addressing a large scale DDoS attack on our outbound network; We’re slowly clearing out the queues from the attack and the latency has been reduced down to normal levels.
Now that the major launch of ExchangeDefender 5 has been completed and we have our processes together I wanted to resume the helpful NOC posts we did in 2009 that gave you a bit of a heads up about maintenance and network changes that were being made. Here is our agenda for March: ExchangeDefender upgrade to Exchange 2010 ExchangeDefender LiveArchive has been tremendously popular and we’ve planned some enhancements to the product that will just blow you away. Coming in March, ExchangeDefender LiveArchive will be upgraded to Exchange 2010. Unfortunately due to the volume of mail, a migration will just not be possible (think restoring or mail moving a year worth of your email multiplied by every ExchangeDefender customer – not easy). To alleviate the pain of getting a fresh mailbox, we will continue to run the two ExchangeDefender systems in parallel – old LiveArchive with Exchange 2007 won’t go offline for 30 days after the new one is up and running. Now let me address a concern:
Now, the beauty of ExchangeDefender 2010 is all in it’s Outlook Web App – no more Web Access - The new solution is compatible with all browsers which means full mobility and a ton more functionality. We’re also building in rule sets so that outbound mail goes to send items and adding a layer of quality assurance that will address the issues we faced with 2007. Rollout of Exchange 2010 Hosting & New Control Panels We will begin offering Exchange 2010 hosting this month as well, with a new quota and a new set of control panels. With the new controls you will be able to manage mail enabled contacts, public folders and distribution groups, so pretty much the same type of control as if you had your own Exchange cluster! As with all new software, the price is going up but we will keep it at the same level for our ExchangeDefender Service providers. We’ll also be the only company to offer it under your own brand while not competing with you for the business – not directly with our brand and not indirectly through the major distributors. With ExchangeDefender we believe we can deliver the reporting and business intelligence, something that is unparalleled in our industry not to mention integrated with Autotask and ConnectWise. Other Fun We’re rolling out new power switches and reboot switches which will make us a lot more green and improve power. There will be several announced reboots / server moves as we move systems around to improve power utilization and cooling. We’ve grown tremendously since ExchangeDefender 5 launched in December 2009 and our products are a hit in the marketplace. ExchangeDefender will be getting a new major data center (delivery point) and we’ll advise of the maintenance as we get closer to the announcement. We also have some really great EU news but look for that in the newsletter coming around the 2nd or 3rd week of March Sincerely. Vlad Mazek, MCSE CEO, Own Web Now Corp
Tonight at 9:00 PM Eastern we will be starting extended maintenance on the Exchange 2007 server DEWEY. During maintenance, access to the DEWEY server will be limited and service is expected to be restored by 11:00PM Eastern. No mail will be lost during the maintenance window. Update 8:00 PM Eastern: We will be moving the maintenance to 8:30 PM to alleviate the latency issue with DEWEY sooner. Update 10:05 PM Eastern: We will be extending maintenance on DEWEY as we migrate users to the newer database. Mail destined for users on DEWEY is being queued by ExchangeDefender and any users needing immediate access to their live mail should login to LiveArchive at https://livearchive.exchangedefender.com using their ExchangeDefender credentials. Update 7:13 AM Eastern: Maintenance and optimization has been completed on DEWEY.
Due to use complaints about access issues, we’ve had to conduct an emergency reboot on DEWEY to increase the resources available to the server. Service is expected to be restored in under 15 minutes. Update 1:52 PM Eastern: The server came up from the first reboot, but the changes did not apply properly. The server is going down for another reboot to attempt to apply the changes.
At 9:40 we experienced a power event in the Dallas DC affecting ExchangeDefender internal infrastructure. This affected performance and availability of admin.exchangedefender.com, this web site, our support portals and Exchange hosting infrastructure. By 9:50 everything appears to be back to normal but as these are highly utilized servers the performance may be affected for another 10-15 minutes as we run our tests to make sure everything is back to normal. We apologize for the inconvenience and will update this ticket shortly.
Tonight (2/22/10) around 6PM Eastern we will begin to implement split GALs on the Australia Exchange 2007 server. Between 6PM and 8PM Eastern there is expected to be brief interruptions in service as the new security policies are implemented.
The Ahsay service on backup74 has been temporarily stopped as we migrate users to the newer volumes. Service is expected to be restored by 4pm Eastern.
The Dewey BES service has expierenced a few random issues since Microsoft patching on the 10th. In an attempt to revert the changes, we’re in the process of removing the updates before restoring from a backup. Service on DEWEY BES will not be interrupted throughout the extended maintenance of the BES server. Update 10:50 AM Eastern: The BES service for Dewey has been restored 100%. We found an issue with two security updates, including KB977914
The Ahsay OBS service on backup74 is currently stopped as we migrate users to larger volumes. Service is expected to be restored by our peak backup time of 12 am Eastern.
At 8 PM Eastern tonight we will be taking backup74 offline to reattempt storage installation. The installation and provisioning is expected to take up an hour and a half and service will be automatically restored after the upgrade is complete. Update 10:07 PM Eastern: The new controller install failed again. We’ve hooked up a temporary drive to alleviate the space issue while we order a new controller. Service is expected to be restored by 11 PM Eastern. Update 10:58 PM Eastern: Service has been restored.
We’ve had reports from a few clients that they are having issues accessing our Exchange 2007 server DEWEY. We are currently investigating into this issue. Update 11:20 AM: Service on DEWEY has been running 100% over an hour. It seems close to 10 PM Eastern on 2/4/2010, IIS had issues compiling the metadatabase..which ultimately did not throw any errors in windows logs or in any of our monitoring software. We ran a quick repair on the metadatabase soon after being notified of service outage and restarted IIS which resolved the issue. |
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
