Last week (January 8th and 9th) we received a dozen reports of messages that simply vanished in the ExchangeDefender system. Upon investigation it turned out that one of the antivirus engines was picking up false positives: marking messages with certain PDF attachments as infected when in fact there was no infection there. The actual infection was simply a detection of an exploit, one that can easily and inadvertently be created by older versions of Acrobat.

We have removed the antivirus engine from the rotation (don’t worry, everything is still being scanned by several other scanners). While the problem in the definition files was already addressed (Exploit.PDF-9669) and widely blogged and discussed, we need a way to deal with false positives. Prior to this we have never had an instance of a reported false positive with an antivirus engine but as more antivirus vendors get into the business of not just detecting viruses and worms but also exploits and other dangerous content, our reporting will have to get better as well.

The bigger question here is: Why was I not notified? If this happened here, it would also explain why I am never received any of the other messages. Allow me to address that in two ways:

1) Almost all of our “missing messages” tickets are related to the messages being quarantined as SPAM and not coming into LiveArchive. At the present time there is no way to get a SPAM message into LiveArchive, even after it’s released from the Quarantine. Because our replication is done at the scan time, we have to move the copying protocol elsewhere to enable post-release and SPAM content.

Followup question: But Vlad, I need to be able to view my SPAM and respond to it while my server is down!! And you can, right from admin.exchangedefender.com! All of our new enhancements are coming to that portal which is completely partner branded and next month we’ll even have training you can just point your clients to.

2) We have never before seen a false positive from an antivirus engine. We’ve seen it crash, we’ve seen it fail to detect a real infection, we’ve seen it bring the scanning node to a crawl and just about everything you’d expect from a piece of security software: just never a false reading. Consequently, we never wrote a process to monitor for the false positives and we never bothered to present the infection logs because so many contained meaningless junk. Several years ago, after countless alerts for Sober and Nimda and so on, we disabled end user reports for antivirus and it was eventually dropped from the product completely.

At this time,  we are sketching a way to put back a configurable alert system for infections should this happen again. We are also creating a system by which you’ll be able (administrators only) to access the infected quarantine items from the web UI).

IMPORTANT: While these infections appeared to be lost forever, we do have them stored on our servers. Reported messages are being released (by hand) by our support teams so if you know the message sender/recipient/subject and date the message was sent, we can retrieve the message and deliver it.

-Vlad

We are currently working on livearchive.exchangedefender.com and the server is currently offline. The IIS application pool keeps crashing since a Windows Update from last night. We expect to have the server up shortly.

Update 9:00 AM Eastern: We’ve resolved the issue with livearchive OWA.

All of our BES servers are currently offline as we move the virtual disks to the new RAID set added in yesterday. All servers are expected to be online in the next 45 minutes.
Update 2:40 PM Eastern: The final BES server is coming online. This will complete the build and migration to the new BES host.

As mentioned last week, we are now deferring all mail from popular SPAM blacklists at SpamCop and SpamHaus. It is important to stress that we are not blocking or rejecting mail from these sites, merely temporarily deferring accepting messages. This subtle difference is what separates spammers from legitimate senders. Legitimate mail server operators will immediately notice they are on an RBL and will address the issue and remove themselves from it.

Our choice of SpamCop and SpamHaus came after years of use, peer reviews and our own statistical models indicating that they rarely make mistakes. We are not using third party reputation lists or greylisting which will delay mail delivery, we are just making sure that the mail arriving to you is from a legitimate source and a secure mail server.

Important Notice: tempfail effect on SureSPAM

Nearly all the messages in SureSPAM quarantine was from SpamHaus and SpamCop. As a result of us tempfailing mail from these known SPAM sources, you will see a significant decrease in SPAM and junk mail report stats as well. If you have clients that you have not yet migrated from SPAM reports to our Outlook and Desktop software, we recommend sending them the following alert:

“Today <Product Name> started temporarily deferring mail from known SPAM sources. This change will make your mail flow more efficient and reduce potentially fraudulent mail (phishing) that slips through when you whitelist large company domains.

As a result of this change you will see a significant reduction in SPAM report contents and will have less SPAM to review through the <Product Name> Outlook addin and/or Desktop agent.

P.S. Please note that we are not bouncing or rejecting mail, we are simply deferring it to allow legitimate mail server operators to address the reason why they ended up on the RBL in the first place. The two blacklists in use are SpamCop (www.spamcop.net) and SpamHaus (www.spamhaus.org) and both provide very reputable databases of known spammers. If you are having an issue receiving mail from certain recipients, have them make sure they are not on known blacklists at the above sites.”

We are closely monitoring the network during this change and will update the NOC blog if there are any issues. We do not expect anything unusual to come as a result of this implementation.

The livearchive server will be going down for a reboot momentarily as we resolve queue related issues.

Updated: 10:47 PM Eastern: After multiple attempts to quickly recover the database, the store still has performance issues mounting. We are going to work out a plan with Microsoft to repair the database and then merge it with the running (new) mailstore. We do not consider any data to be lost. We have 6 TB of a mailstore to repair. We estimate that it could take up to two weeks to merge the data together.

Updated 4:09 PM Eastern: We’ve restored mail flow and login ability to the server, however, the old mail store is dismounted and being repaired. We plan on merging the current mailstore that is actively running and the old mail store.

Updated 2:43 PM Eastern: We have been working with Microsoft for the past hour to resolve the mail delivery and login issue to LiveArchive. There is no current ETA on resolution time.

We are investigating issues with hosted exchange 2007 servers in Dallas at this moment. The service is currently being restored and should be back to normal momentarily.

A number of users have reported inability to receive email. Sending email works fine but receiving does not. In multiple tests against multiple sites, both protected by ExchangeDefender and not, we have seen nearly 60% connection drops.

At this point we believe this is related to the latest Microsoft patches – and a simple reboot appears to clear out the issue of no port 25 connectivity. Don’t worry, ExchangeDefender is holding your email. It does not appear to be fixed by removing the patches and reapplying them.

As our friend Susan Bradley just told me via IM: “Full moon passing”

Vlad Mazek, MCSE

CEO, Own Web Now Corp

Good morning – we are currently investigating two blacklists from large ISPs targeting one IP address on the ExchangeDefender network. Two services are identified as Verizon and FrontBridge and we have opened requests to be removed from both along with any data that might help us find out why the lists were put in place to begin with given our 0 tolerance for SPAM. The rejections are marked by:

< mail157-dub-R.bigfish.com #5.0.0 X-Postfix; host    winse-6216mail6.customer.frontbridge.com[131.107.115.215] said: 550 5.7.1    External client with IP address 65.99.255.232 does not have permissions to  submit to this server. Visit http://support.microsoft.com/kb/928123 for more information. (in reply to end of DATA command)>

 

<outbound2.exchangedefender.com #5.5.0 SMTP; 571 Email from 65.99.255.232 is currently blocked by Verizon Online’s anti-spam system. The email sender or Email Service Provider may visit http://www.verizon.net/whitelist and request removal of the block.>

In the meantime, you can change your outbound smarthost to outbound1.exchangedefender.com if you experience the problems above.

On the funny side: We find it hilarious that Microsoft is linking to their own KB articles in the rejection note for a problem that is caused on their own servers – how about something more helpful like postmaster or delisting contact address or URL! Even more surprising is that Microsoft FrontBridge is running on an open source Postfix mail platform, not a Microsoft one.

We will update you on the delisting process as we get more information. The problem should not impact many senders as 65.99.255.232 is just one of the nodes in our outbound network.

In about 10 minutes (11 CST, -6 GMT) we will be shutting down the Offsite Backup infrastructure for the memory and networking upgrades to accommodate the new global replication platforms and proxies. The maintenance interval is expected to last less than an hour and we do not expect anything out of the ordinary.

For approximately one hour between 2PM and 3PM EST our phone service was knocked out. Due to growth we ordered a block of additional DIDs (direct dial in numbers) and our VoIP IAX provider inadvertently reset all our DIDs to a different proxy server. Unfortunately, we were registering our PBX to the wrong VoIP proxy server and as a result of it, majority of our direct dialin numbers failed to connect. Among major numbers affected were our primary toll free number (877) 546-0316 and our primary local number for international calls (407) 465-6800.

We were able to quickly identify and resolve the problem but suffered approximately an hour of outage without a record of phone calls missed. We are sorry about any inconvenience this may have caused, if you missed us please keep in mind that you can reach most of us via the support portal, email, instant messaging and of course, direct sip dial if you are on VoIP as well we do accept connections at sip.ownwebnow.com  

We are experiencing significant issues with the virtual hosting of Microsoft Windows 2003 servers. We are still collecting data and trying to isolate the issue but some virtual servers simply stop responding to the network requests – either before or after the latest Microsoft security updates. To further complicate issues, virtual machines are crashing upon reboot requiring two reboot cycles.

We are still working on a solution, it appears that after the second reboot the system remains stable and accepts network connections without an issue. Please stand by for update, as of 2:00 PM CMT the entire network appears to be completely operational.

Earlier today we identified an issue with one of the nodes in the ExchangeDefender admin cluster that was set with the wrong clock (wrong timezone configuration) and although the times were in sync across all servers, due to many scripts relying on unix timezone settings to calculate offsets and localized time, some users have experienced timestamps that were off by one hour (-1)

This issue was resolved at roughly 8:30 AM and while it did not pose a huge issue we felt it was important to note for any of you that may have noticed weird times in the admin logs.

We have officially frozen all external development tasks until November 10th, 2007. All timelines will be adjusted two weeks from today to accommodate internal development and work on projects related exclusively to Own Web Now Corp. You should not see any new features nor the changes to the default system behavior of any of our products until November 10th, 2007.

Thank you.