We have been running an extended maintenance interval process on ExchangeDefender to run some optimization processes with regard to PSA integration. Some users have reported that they were unable to access admin.exchangedefender.com, something we have no explanation for at the moment as we’ve not had any service interruption on it (just high loads and periodic latency).

Our Exchange 2007 network is also under extended maintenance due to the issues we’ve experienced between Exchange 2007 Update Rollup 9 and Blackberry Enterprise Services. We are still working on it and will provide updates as we have them.

We’ve received complaints from multiple partners that ExchangeDefender reports were not sent at the previously scheduled time. We are investigating the cause and working towards a resolution.

We anticipate receiving reports about delays in ExchangeDefender for mail accepted in the earlier half of today. In short, the livearchive server experienced a larger than normal queue load, which delayed messages from the inbound grid being copied to livearchive.

This issue has been resolved and any queued mail should be delivered momentarily.

There have been a few reports about NDR error “ExchangeDefender does not protect this address” on new aliases/accounts added to ExchangeDefender today. The NOC has identified the issue with replication of new accounts and applied a fix. Accounts should start working properly by 6:30 PM Eastern.

Over the weekend our reports service was under maintenance and reported all zeros. This issue has been addressed, things should be back to normal within an hour or so. I wanted to take a moment of your time and address the email reports again.

As usual, the 8 partners that have clients using this service complained, loudly. I have talked about this numerous times and heard the feedback but the fact remains that the life cycle for this feature has come and passed. Email reports are not timely, are not realtime, get caught by other filtering software due to their content and admin’s improper deployment, crash Outlook when they report thousands upon thousands of SPAM messages (volume of which is still growing), and are naturally ignored by over 99.9% of our client base.

If you have sold your client base or your employees on the email reports please inform them of the other, better, methods to access their junk mail. With ExchangeDefender 5 we will deliver a cross-platform desktop agent and a more responsive web UI, completely eliminating the need for digest mail reports. I know many of you will miss them, and they were a great solution for SPAM reporting back in early 2000’s, but with users receiving tens of thousands of messages a day this process is no longer supportable.

Please consider the following:

ExchangeDefender Client Software

ExchangeDefender Desktop Shortcut

Lastly, I would like to address a concern that a few resellers have brought to me regarding the “email reports as a competitive advantage” which may work against appliances but is still several levels below the functionality and value that ExchangeDefender delivers with the client software suite. We designed this software to give our partners and clients a huge gateway to the users desktop while allowing them to remain in the workplace environment they are used to. If you are not leveraging, branding, deploying and marketing these tools you are missing an incredible opportunity to demonstrate the value of the service you are providing – and price it accordingly.

Sincerely,
Vlad Mazek, MCSE
CEO, Own Web Now Corp

We are experiencing an issue with the ExchangeDefender email SPAM reports: they are reporting all 0’s. Reporting will be restored within the hour.

As usual, we do not recommend or offer support for email reports as it is a feature that has long ago been replaced with more reliable and efficient ways of obtaining SPAM data and statistics through either our administrative portal, ExchangeDefender Outlook 2007 plugin or ExchangeDefender Desktop Agent. If you have not considered these alternatives please take a look at: http://www.exchangedefender.com/features_client_software.php

Earlier last week we identified an issue with ExchangeDefender whitelists. At first we believed the issue was simply that of certain notes not receiving full replication but after that turned out not to be the case we focused on the internal processes that generate note whitelists. The problem turns out to be a bug in the replication of whitelist content between users and master whitelist.

We have been working on it through our maintenance interval but have added a few more hours for additional testing today to make sure it is working properly. As a result of the bug we have rewritten how the whitelist data is replicated and as a result monitoring scripts that watch for replication failures had to be adjusted as well.

As of 1 AM EST / 7 AM GMT the monthly global maintenance cycle has been completed. Please allow up to 2 hours for routes to converge and new networks to be announced.

Among updates, features and bug fixes are:

- Provisioning for the new Los Angeles, CA and Chantilly, VA networks.

- Expansion of LiveArchive network.

- Fix for the ExchangeDefender routing network priority hack (for low bandwidth mail servers)

- Fix for Australia/Pacific routing group (* extending smtp_greet from 30 to 120 seconds to compensate for trans-Pacific)

- Extended nameserver network to improve load balancing.

- Provisioning for the new RR routing strategy for international customers with complex compliance issues (ex: “route all mail through UK, route through EU in case of network outages, never route outside of EU”)

It will take at most two hours for the routes to converge, during this time you may see delayed messages or deliveries out of sequence as the new paths are being distributed. You also may see delays in extended-outage-queues we use for disaster recovery (ETRN/queueing/spooling) mechanisms which are affected by all of the above but primarily the name server expansion.

These efforts are critical to the continued growth and reliability of the ExchangeDefender network. No downtime or outage has occurred.

We are currently running an experimental delivery process on ExchangeDefender mail that has been queued on ExchangeDefender delivery queue for over an hour. If our management system indicates that the target server is up and the message is continuously deferred/rejected, the message will be handed off to another delivery agent that will attempt to stream it through a different process.

To review, if the following conditions are met:

1. Recipients mail server is up and running
2. Recipients SMTP port is available and accepting connections (no greylisting).
3. Message is older than an hour and smaller than 100 Mb.

.. our system will attempt to flush the message through a different system.

These messages may be malformed in some way and the recipients server never would have received them directly from a third party server. However, with ExchangeDefender in the middle, we believe we can attempt to compensate for the problems in the message.

What to watch out for:

Our redelivery queue currently stands at 14 days. It is possible that you will see messages that are older than just a few days.

Why are we doing this:

We have decided to get to the bottom of the sporadic issues that have been reported through the years and build a concise, managed system that monitors for clients server problems so support requests can have a more timely and adequate response. We will extend this Managed ExchangeDefender service as we go along.

Yesterday we made performance enhancements to our ExchangeDefender Email Reports to help cope with the amount of junk arriving in reports as of late. As of this morning, it still does not show signs of improvement we have been expecting. While we are still trying to address the issue, we do not recommend using ExchangeDefender SPAM reports and suggest moving to Outlook or Desktop agents for ExchangeDefender or preferably the web interface.

Update: We believe the issue has been resolved and is now behaving correctly. We are continuing to watch it.

Symptoms: Empty intraday, fractional daily reports.

Update: We are currently investigating performance problems from 5 AM – 8:20 AM on our outbound network. One of the load balancers handling the outbound mail failed, creating a performance issue on the other server and backing up customers SMTP queues due to load throttling. No mail was lost.

This issue has been resolved at 8:00 AM EST but is being monitored further as the systems catch up and sync up.

Earlier this morning you may have received the following error while relaying through ExchangeDefender:

< outbound2.exchangedefender.com #5.7.1 SMTP; 550 5.7.1 … Access denied to 65.99.255.236 by new.spam.dnsbl.sorbs.net DNSBL (http://dnsbl.sorbs.net/)>

This issue has been corrected and mail is flowing through correctly. If you received this error please resend the message as it was permanently rejected by the recipients mail server. 

AT&T is having RBL issues again, we are working with them to resolve the problem. You may receive this problem when emailing the AT&T network for the time being:

<<< 521-65.99.255.236 blocked by sbc:blacklist.mailrelay.att.net.

<<< 521 DNSRBL: Blocked for abuse. See http://att.net/blocks

554 5.0.0 Service unavailable

We have put in place a workaround and are working with AT&T to resolve the issue. You should not continue to see this problem. However, the issue is still open.

We will be conducting a major maintenance window this Sunday, November 8th, 2008.

We will be deploying series of hotfixes provided by Microsoft for a slew of bugs Own Web Now Corp has reported over the past six months. We have also received a lot of guidance in the way of optimizing our setup and will with Microsoft’s help proceed to make major adjustments to the platform.

Unfortunately, this means that some users may experience issues during Sunday early AM hours. Although our systems are clustered some changes require database moves and service restarts which will have to be done in sync and will unfortunately lead to service interruptions.

Our goal as always is to keep these service interruptions to the minimum and limit them to maintenance hours, however, since these issues will be sporradic throughout the night we wanted to note them here.

After the initial test on our own Exchange 2007 network we will be applying the same fixes and optimizations to our dedicated server clients running Exchange 2007.

ExchangeDefender will not be impacted, however, your mail may experience slight delay if you are on Exchange 2007 mailbox store which is being cycled and ExchangeDefender is not able to immediately deliver the message. In this case we recommend all our mission critical 24/7 operations to fall over to LiveArchive which will be available.

Over the past 12 months we have had a 99.999% uptime on our Exchange 2007 network and 100% uptime on our ExchangeDefender network. Those numbers are impressive but only possible thanks to preventive maintenance and optimizations as noted above. We apologize in advance for any inconvenience you experience during the maintenance cycle.

We have been made aware of an issue with email reports showing all 0’s for SPAM stats. The issue has been resolved as of 11:30 AM EST.

Please note that we do not recommend using email reports and encourage everyone to migrate to the new methods of accessing SPAM: realtime web portal, desktop agent or Outlook 2007 agent.

At approximately 4AM EST we have noticed a failure in updates from one of our AV vendors. That failure produced higher than expected virus matches which ended up queuing a larger than normal amount of messages. We have resolved the issue with the update and are currently re-processing all the mail that was quarantined over the past few hours.

Please stand by, we will deliver all mail.

Update: 7:29 AM EST: Nearly all the mail that was affected by the faulty AV update has been processed and has been dispatched to delivery queues. As of the previous update, all new mail has been delivered in realtime. It is important to note that we are only processing the backlog for the messages that did get trapped by the faulty AV update.

Update: 9:15 AM EST: 99% of the messages have been flushed out. By the time you read this posting all the mail would have been delivered. No mail has been dropped during the period, if you experience further issues with delays please follow our deployment guide and support documentation, we find most delays are related to the on-premise issues relating improper firewall configuration, connection rate limiting (by far) and other SPAM/malware scanning that does not properly whitelist ExchangeDefender systems.

As you may have noticed over the past few weeks, the SPAM levels have increased slightly. Unfortunately, even a slight increase in the SPAM levels as a percentage can result in getting a piece or two an hour as opposed to a piece or two a day. Yesterday we finally isolated the issue that was causing this thanks to a few of our partners and the new ExchangeDefender Outlook 2007 addin. We are still working on automating the distribution and monitoring of the new processes that will keep this from coming up again.

Further Details

ExchangeDefender has multiple grids around the world. All grids use a central RBL distribution database that is centrally managed and monitored. Every grid has it’s own DNS caching servers that hold both the RBL data as well as our clients IP address information for delivery, routing and SPAM definitions. Since the latest update to our core distribution the DNS server performance has been flaky and would simply stop returning results. Because our RBL code is set to look for matches in the RBL zone the servers lack of response, or lack of correct response, means that the messages that were certainly SPAM were allowed to go through the less-restrictive SPAM scanning and unfortunately that contributes to 1-2% difference in the SPAM load and in some cases latency for nodes that are about to go into the shutdown/maintenance mode and are flushing out their queues. Because ExchangeDefender delivery queues run off the same DNS infrastructure (technical limitation) this compounds the problem and issues as the resolutions do not come from the primary (on-node) or secondary (on-grid) but a tertiary (central OWN NOC) DNS server.

What we have done so far is implementing a system that does local resolver check and restarts the DNS service if it is not returning proper data.

What we are currently working on is a monitoring system to centrally report the issues with the resolver latency (one of the things we currently do not measure) as the lookups have to skip to the secondary or tertiary systems.

We expect to have all the issues handled by the end of the weekend. From statistical breakdowns we know that the issue has not been widespread (only certain users would even have noticed the difference) and only about a dozen people have complained so far. Unfortunately for us, the people likely to notice are the people that get the most mail and the ones that likely love our product the most. We’ll get this one taken care of for you folks, thanks for your patience.

We have received several reports of issues with BT. You may receive this error when sending messages to btinternet.com recipients.

The e-mail system was unable to deliver the message, but did not report a specific reason. Check the address and try again. If it still fails, contact your system administrator.

< outbound2.exchangedefender.com #5.0.0 SMTP; 554 <user@domain.com>: Relay access denied>”

We have notified BT by e-mail and phone regarding the issue, the problem is on their end. Since this is a configuration issue on BT network we have no ETA, no resolution time or idea of what may be going wrong.

For more information about proxy errors, click here.

We are currently tracking issues that have been reported by multiple users:

1. Email reports for ExchangeDefender SPAM quarantines are not being delivered to the users that have been configured to receive them. So far we have narrowed it down to the 00:00 EST time reporting interval for daily reports. We will know more about this around midnight.
2. Offsite Backup reports are not reaching some clients. We are working with AhSay to isolate the issue and will likely be applying a hotfix later in the day. This is not a widespread issue either but we are taking it seriously since it has been reported multiple times.

We will update as we get more information.

We will be extending our maintenance window for the report services this weekend in order to implement the new ExchangeDefender 4.0 functionality. While the reporting should not be impacted during this time, our support teams will have limited visibility to the backend and might not be able to effectively troubleshoot the issues. We are sorry for any inconvenience this might cause your clients but we’re confident you will be pleased with the results.

Maintenance: Sunday, 1 AM EST – 6 AM EST.