We had an authentication issue reported on daisy.theofficeserver.com Exchange 2003 hosting platform. The issue was related to an older SSL certificate binding to the wrong IP address and stopping authentication and OWA requests that check for the certificate validity.

Issue was promptly corrected and old certificate permanently removed.

Thank you and have a great day.

We will be conducting a major maintenance window this Sunday, November 8th, 2008.

We will be deploying series of hotfixes provided by Microsoft for a slew of bugs Own Web Now Corp has reported over the past six months. We have also received a lot of guidance in the way of optimizing our setup and will with Microsoft’s help proceed to make major adjustments to the platform.

Unfortunately, this means that some users may experience issues during Sunday early AM hours. Although our systems are clustered some changes require database moves and service restarts which will have to be done in sync and will unfortunately lead to service interruptions.

Our goal as always is to keep these service interruptions to the minimum and limit them to maintenance hours, however, since these issues will be sporradic throughout the night we wanted to note them here.

After the initial test on our own Exchange 2007 network we will be applying the same fixes and optimizations to our dedicated server clients running Exchange 2007.

ExchangeDefender will not be impacted, however, your mail may experience slight delay if you are on Exchange 2007 mailbox store which is being cycled and ExchangeDefender is not able to immediately deliver the message. In this case we recommend all our mission critical 24/7 operations to fall over to LiveArchive which will be available.

Over the past 12 months we have had a 99.999% uptime on our Exchange 2007 network and 100% uptime on our ExchangeDefender network. Those numbers are impressive but only possible thanks to preventive maintenance and optimizations as noted above. We apologize in advance for any inconvenience you experience during the maintenance cycle.

Over the past week or so we have been tracking sporadic complaints about mail delivery delays to Exchange 2007 hosting. It appears that in rare cases the Exchange 2007 network temporarily rejected the message with a 4.3.1: Insufficient system resources message even though plenty of resources were available on the system to process the message and plenty of storage was available on the volume.

We have seen this issue creep up from time to time on multiple Exchange 2007 issues and have provided information to Microsoft regarding a possible bug but have so far been unsuccessful in getting a response because the issue is so sporadic that it cannot be pinned down to a single factor and only a small number of messages encounters an issue. What is even more frustrating is that even during heavy maintenance and optimization tasks the issue does not creep up where it is expected with back pressure controls. This Sunday we ran a database optimization maintenance process and out of roughly 1,000 messages processed less than 20 were deferred with the Insufficient system resources error even though the CPU utilization was above 80%.

Due to the level of complaints and our inability to isolate this issue with Microsoft we have disabled the back pressure feature of Microsoft Exchange 2007 on all our systems until further notice.

Further details about Exchange 2007 and back pressure is available at Microsoft TechNet.

Note: This issue is isolated to the new feature in Exchange 2007 called back pressure. Exchange 2003, ExchangeDefender and ExchangeDefender LiveArchive are not affected by this issue.

Following is our reboot cycle for our global Exchange 2007 network. Microsoft recently published the Exchange Server 2007 SP1 Update Rollup 3, SQL Server Service Pack 2 and several security patches that have been tested by OWN and approved for rollout. Scheduled reboots are as follows:

USA - 5 AM EST (11:00 GMT) Today

Europe - 5 AM GMT Tomorrow

Australia - Midnight EST-Australia Today (10 AM EST United States)

We do not anticipate any outage or issues with the patches, they have passed lab tests without problems.

We are currently addressing an issue with Exchange 2007 OWA. You will see the following text when attempting to login:

Outlook Web Access did not initialize. An event has been logged so that the system administrator can resolve the issue. Please contact technical support for your organization.

Request
Url: https://scrooge.exchangedefender.com:443/owa/auth/error.aspx

We will have this addressed momentarily and update this site.

Update: 2:45: Problem has been solved.

We will be starting the SP1 upgrade on our systems in roughly two hours (10 AM EST, -5 GMT) and expect to have all operations completed by noon. To minimize the surprises and potential conflicts, the entire cluster and all its members will be patched at once. That unfortunately does mean a bit of total outage, but it does minimize the chance of anything breaking in the process. Hosted Exchange has been one of our most solid products and we look forward to keeping it that way.

Update: Exchange 2007 Service Pack 1 is now up and running, systems have been imaged and we are all done.

The following is an update on the extended network maintenance on the ExchangeDefender networks spanning both ExchangeDefender Exchange Hosting network and ExchangeDefender SMTP Security networks. The work continues, full details on the changes, upgrades and enhancements will be published Monday:

Completed network maintenance tasks:

1. Full implementation of split queue and RBL prioritization and resource weights completed.
2. Throttle control of delivery queues completed.
3. Sender watermarking completed.
4. New AS engine and SPAM policy handling completed (99.1% effectiveness, 0 false positives against the new rules, lowest complaint rate at SPAM levels in history of the product, woohoo!)
5. Slave server database management tools and self-healing systems online.
6. Additional ExchangeDefender nodes online in Dallas and Los Angeles.
7. Load balancer enhancements including traffic shaping of spamming networks through the ExchangeDefender staging systems for R&D completed.
8. Enahnced logging for NDR, Reject, Malformed Headers, Data Format Errors and more.
9. Network latency problems fixed.

Remaining tasks for Sunday:

1. Microsoft Exchange 2007 SP1 rollout
2. ExchangeDefender split queue delivery handlers with skipahead enabled
3. Provisioning and testing of EU LiveArchive

on Wednesday (1 AM - 3 AM CMT, 6:00 - 9:00 GMT) we will be provisioning the new reporting server infrastructure and full integration into the ExchangeDefender core network/systems. This will allow us (and you) to directly manage the behavior of each filter on a granular level.

More on this on Monday.

At approximately 5:48 EST several of our large customer systems started a systematic DDoS against our outbound network. Because these are trusted systems, they are allowed access to our network without many checks. Unfortunately, these customer systems have been compromised, leading to a DDoS launch of close to 50,000 messages per second.

We have removed offending systems from our grid and mail is flowing normally. Unfortunately, outbound messages relayed during this time period of 5:48 - 6:20 EST (13:48 - 14:20 GMT) were destroyed or mangled. If you have sent messages during this time period, please resend.

We are sorry about the inconvenience and we have acted in as fast of a response as we could to what was a rather massive DDoS against our outbound network.

During early morning (2-5 AM) some customers on archive.exchangedefender.com were affected by a permission rebuilding script that failed to complete permission resets on the mail folders. The bounceback looked like this:

Diagnostic-Code: X-Postfix; maildir delivery failed: create maildir file /home/user/tmp/maildir: Permission denied

We have since fixed the issue and appologize for the inconvenience this may have caused.

We are currently addressing a licensing issue on one of our Microsoft Exchange 2003 shared hosting servers. If you are on donald.theofficeserver.com server, you will not be able to login to the server. The deployment had apparently lost all its licensing and defaulted to the Windows Standard 5CAL pack for some reason. We are working on addressing this right now, please stand by for update.

Update: Issue solved

We were able to restore the licensing information from backup and resume normal operations after a reboot.

All services are back to normal, no mail was lost during the reboot cycle or licensing issue as ExchangeDefender serves as a system failover.

Earlier today the SSL certificate on our Exchange Hosting servers (daisy) expired and we were not able to renew the certificate in time. We were forced to obtain the certificate from another authority that was more responsive and SSL service has been restored to a certificate that properly validates. If you encountered any issues with certificate warnings or RPC-over-HTTPS this morning it was due to this issue, everything is back to normal right now.