Tonight starting at 8:30 PM Eastern we will begin upgrading the DEWEY Exchange 2007 network by installing additional domain controllers and additional hub transport servers. During the upgrades, we will have to reboot the mailbox and primary DC for DEWEY. Users may experience brief periods of disconnection from the Exchange server during the upgrades.

Update 8:35 PM Eastern: We are about to reboot the primary mailbox server for DEWEY. Connectivity is expected to be degraded for up to 20 minutes.

We are beginning the process of continuing the upgrades to HUEY and Scrooge. We are beginning again with HUEY and will move on to Scrooge. Just as yesterday, access to mailboxes through Exchange will be interrupted until the upgrade is complete. Each server is scheduled to take about 4 hours.

Ref: http://www.ownwebnow.com/noc/2010/08/23/huey-and-scrooge-upgrades/

Update 1:37 AM Eastern: The imaging of the OS on HUEY was successful. We are in the process of moving the database and beginning work on Scrooge

Update 3:18 AM Eastern: The DB copy on HUEY is still in progress and is about half way completed. Since the information is being written to a new RAID volume, the copy process must write to all disks in the array. Scrooge has about 35 minutes on the OS image.

Update 4:42 AM Eastern: The DB copy has begun on HUEY and windows estimates 2.5 hours for completion. We are about to begin the image of Scrooge

Update 5:00 AM Eastern: The DB copy completed on HUEY and service has been restored.  Scrooge is committing the backup image to the new server and should be ready for the database copy in 30 minutes.

Update 7:44 AM Eastern: All upgrades were a stunning success. The upgrade to HUEY is complete and the databases were moved. Scrooge is on the new hardware, however the databases copy was taking longer than our allowed threshold. We’ve moved the old database drives to the new server and will complete the DB move either tonight or over the weekend.

Service is expected to be operational by 8AM Eastern, but may be slightly reduced until 8:30 as both servers process pending mail.

Update 8:08 AM Eastern: As a precautionary action we are running a chkdsk on the Scrooge DB drives to ensure they will be able to serve clients without issue. The scan is already 30 minutes in and should be completing by 9AM.

Update 8:55 AM Eastern: We are going to let the chkdsk run until 9:30 AM Eastern before we look into switching to the backup drive. We appreciate our partners patience as this upgrade will provide clients with a smoother Exchange experience.

Update 9:25 AM Eastern: Running a quick reboot in HUEY to correct the system time clock

Update 9:32 AM Eastern: The reboot on HUEY completed and the information store is coming online. If any clients experience mail “disappearing” after it shows up as new, please check the mail from Yesterday  in Outlook as the time on the server was one day behind.

Update 10:00 AM Eastern: The chkdsk on the drive is on the final stage, currently at 30%. The server should be 100% operational by 10:30 AM. Canceling the chkdsk was not optional as the other drive presented the same issue as the original.

Update 10:22 AM Eastern: The chkdsk is currently at 50%. This changes the final time to a more realistic 11AM. Customers needing instant access to their mail should utilize LiveArchive until the upgrades are complete. The upgrades were not scheduled to last past 9AM, however we needed to complete the physical migration today.

Update 10:58 AM Eastern: chkdsk is currently at 80% and looks like the slow down was due to a couple logical corrupted free-space allocation blocks.

Update 11:50 AM Eastern: The physical upgrade is complete and service is 100% operational.

Around 4:02 PM Eastern we started receiving alerts from our monitoring system about packet loss on the network. Upon investigation, we found an IP that was flooding with fragmented packets..the ICS disabled the talker, but the backlog of traffic caused some unavailability to clients. 

Access to livearchive will be temporarily unavailable for some users as we dismount databases to make schema level changes for our new monitoring. Service is expected to be fully operational by noon Eastern.

Update 12:00 PM Eastern: Service to livearchive is back to normal operation

Update 1:54 PM Eastern: We’re receiving reports that some clients cannot access livearchive. We are actively investigating the issue.

Update 2:48 PM Eastern: As part of our investigation, we have to disable mail routing to livearchive. Customers should expect to see a gap in mail in their livearchive mailboxes.

Update 3:09 PM Eastern: We are installing more memory modules into the primary mbox server for livearchive. There is no ETA as of yet.

We are in the process of installing Exchange 2007 SP3 to the EUROPE and DEWEY server. Previous attempts to install the SP failed until a resolution was discovered earlier today. Clients may experience a brief disconnection from outlook, but service will resume in less than 20 minutes.

Throughout the night we will be performing upgrades to the backup system for the 2010 Exchange LOUIE network. Users may experience brief delays when interacting with their mailbox, but no mail will be lost during this upgrade.

If a user becomes disconnected from outlook, advise them to either use OWA (https://cas.louie.exchangedefender.com/owa) or LiveArchive (https://livearchive.exchangedefender.com)

The projected window is from 11pm EST – 3am EST. We will upgrade this post in the event of any scheduled outages.

At 9:00 PM Eastern tonight (6/22/2010) we will be installing the long awaited Exchange 2010 Rollup 4 patch which brings back the ability for PST imports through automation. Service is expected to be lightly impacted between 9:00 PM – 9:30 PM as we patch all seven servers.

Update 6/23/10: The patch rollup installation was delayed last night and moved to today. We will begin the patch process at the same time, 9:00 PM Eastern.

Update 10:47 PM: The final patch has been installed and the final reboot is commencing. Service is expected to be 100% operational by 11 PM Eastern.

Earlier this morning we had an outbreak of spam sent through the ExchangeDefender outbound grid. As a result, one of our IPs was listed on an RBL. As a temporary solution, we’ve disabled two of the smart host nodes as we isolate and remove the spam messages. There may be a minor delay in outbound emails, but we anticipate being back to 100% operation by 2:30 PM Eastern.

Last week (January 8th and 9th) we received a dozen reports of messages that simply vanished in the ExchangeDefender system. Upon investigation it turned out that one of the antivirus engines was picking up false positives: marking messages with certain PDF attachments as infected when in fact there was no infection there. The actual infection was simply a detection of an exploit, one that can easily and inadvertently be created by older versions of Acrobat.

We have removed the antivirus engine from the rotation (don’t worry, everything is still being scanned by several other scanners). While the problem in the definition files was already addressed (Exploit.PDF-9669) and widely blogged and discussed, we need a way to deal with false positives. Prior to this we have never had an instance of a reported false positive with an antivirus engine but as more antivirus vendors get into the business of not just detecting viruses and worms but also exploits and other dangerous content, our reporting will have to get better as well.

The bigger question here is: Why was I not notified? If this happened here, it would also explain why I am never received any of the other messages. Allow me to address that in two ways:

1) Almost all of our “missing messages” tickets are related to the messages being quarantined as SPAM and not coming into LiveArchive. At the present time there is no way to get a SPAM message into LiveArchive, even after it’s released from the Quarantine. Because our replication is done at the scan time, we have to move the copying protocol elsewhere to enable post-release and SPAM content.

Followup question: But Vlad, I need to be able to view my SPAM and respond to it while my server is down!! And you can, right from admin.exchangedefender.com! All of our new enhancements are coming to that portal which is completely partner branded and next month we’ll even have training you can just point your clients to.

2) We have never before seen a false positive from an antivirus engine. We’ve seen it crash, we’ve seen it fail to detect a real infection, we’ve seen it bring the scanning node to a crawl and just about everything you’d expect from a piece of security software: just never a false reading. Consequently, we never wrote a process to monitor for the false positives and we never bothered to present the infection logs because so many contained meaningless junk. Several years ago, after countless alerts for Sober and Nimda and so on, we disabled end user reports for antivirus and it was eventually dropped from the product completely.

At this time,  we are sketching a way to put back a configurable alert system for infections should this happen again. We are also creating a system by which you’ll be able (administrators only) to access the infected quarantine items from the web UI).

IMPORTANT: While these infections appeared to be lost forever, we do have them stored on our servers. Reported messages are being released (by hand) by our support teams so if you know the message sender/recipient/subject and date the message was sent, we can retrieve the message and deliver it.

-Vlad

We are currently working on livearchive.exchangedefender.com and the server is currently offline. The IIS application pool keeps crashing since a Windows Update from last night. We expect to have the server up shortly.

Update 9:00 AM Eastern: We’ve resolved the issue with livearchive OWA.

All of our BES servers are currently offline as we move the virtual disks to the new RAID set added in yesterday. All servers are expected to be online in the next 45 minutes.
Update 2:40 PM Eastern: The final BES server is coming online. This will complete the build and migration to the new BES host.

As mentioned last week, we are now deferring all mail from popular SPAM blacklists at SpamCop and SpamHaus. It is important to stress that we are not blocking or rejecting mail from these sites, merely temporarily deferring accepting messages. This subtle difference is what separates spammers from legitimate senders. Legitimate mail server operators will immediately notice they are on an RBL and will address the issue and remove themselves from it.

Our choice of SpamCop and SpamHaus came after years of use, peer reviews and our own statistical models indicating that they rarely make mistakes. We are not using third party reputation lists or greylisting which will delay mail delivery, we are just making sure that the mail arriving to you is from a legitimate source and a secure mail server.

Important Notice: tempfail effect on SureSPAM

Nearly all the messages in SureSPAM quarantine was from SpamHaus and SpamCop. As a result of us tempfailing mail from these known SPAM sources, you will see a significant decrease in SPAM and junk mail report stats as well. If you have clients that you have not yet migrated from SPAM reports to our Outlook and Desktop software, we recommend sending them the following alert:

“Today <Product Name> started temporarily deferring mail from known SPAM sources. This change will make your mail flow more efficient and reduce potentially fraudulent mail (phishing) that slips through when you whitelist large company domains.

As a result of this change you will see a significant reduction in SPAM report contents and will have less SPAM to review through the <Product Name> Outlook addin and/or Desktop agent.

P.S. Please note that we are not bouncing or rejecting mail, we are simply deferring it to allow legitimate mail server operators to address the reason why they ended up on the RBL in the first place. The two blacklists in use are SpamCop (www.spamcop.net) and SpamHaus (www.spamhaus.org) and both provide very reputable databases of known spammers. If you are having an issue receiving mail from certain recipients, have them make sure they are not on known blacklists at the above sites.”

We are closely monitoring the network during this change and will update the NOC blog if there are any issues. We do not expect anything unusual to come as a result of this implementation.

The livearchive server will be going down for a reboot momentarily as we resolve queue related issues.

Updated: 10:47 PM Eastern: After multiple attempts to quickly recover the database, the store still has performance issues mounting. We are going to work out a plan with Microsoft to repair the database and then merge it with the running (new) mailstore. We do not consider any data to be lost. We have 6 TB of a mailstore to repair. We estimate that it could take up to two weeks to merge the data together.

Updated 4:09 PM Eastern: We’ve restored mail flow and login ability to the server, however, the old mail store is dismounted and being repaired. We plan on merging the current mailstore that is actively running and the old mail store.

Updated 2:43 PM Eastern: We have been working with Microsoft for the past hour to resolve the mail delivery and login issue to LiveArchive. There is no current ETA on resolution time.

We are investigating issues with hosted exchange 2007 servers in Dallas at this moment. The service is currently being restored and should be back to normal momentarily.

A number of users have reported inability to receive email. Sending email works fine but receiving does not. In multiple tests against multiple sites, both protected by ExchangeDefender and not, we have seen nearly 60% connection drops.

At this point we believe this is related to the latest Microsoft patches – and a simple reboot appears to clear out the issue of no port 25 connectivity. Don’t worry, ExchangeDefender is holding your email. It does not appear to be fixed by removing the patches and reapplying them.

As our friend Susan Bradley just told me via IM: “Full moon passing”

Vlad Mazek, MCSE

CEO, Own Web Now Corp

Good morning – we are currently investigating two blacklists from large ISPs targeting one IP address on the ExchangeDefender network. Two services are identified as Verizon and FrontBridge and we have opened requests to be removed from both along with any data that might help us find out why the lists were put in place to begin with given our 0 tolerance for SPAM. The rejections are marked by:

< mail157-dub-R.bigfish.com #5.0.0 X-Postfix; host    winse-6216mail6.customer.frontbridge.com[131.107.115.215] said: 550 5.7.1    External client with IP address 65.99.255.232 does not have permissions to  submit to this server. Visit http://support.microsoft.com/kb/928123 for more information. (in reply to end of DATA command)>

 

<outbound2.exchangedefender.com #5.5.0 SMTP; 571 Email from 65.99.255.232 is currently blocked by Verizon Online’s anti-spam system. The email sender or Email Service Provider may visit http://www.verizon.net/whitelist and request removal of the block.>

In the meantime, you can change your outbound smarthost to outbound1.exchangedefender.com if you experience the problems above.

On the funny side: We find it hilarious that Microsoft is linking to their own KB articles in the rejection note for a problem that is caused on their own servers – how about something more helpful like postmaster or delisting contact address or URL! Even more surprising is that Microsoft FrontBridge is running on an open source Postfix mail platform, not a Microsoft one.

We will update you on the delisting process as we get more information. The problem should not impact many senders as 65.99.255.232 is just one of the nodes in our outbound network.

In about 10 minutes (11 CST, -6 GMT) we will be shutting down the Offsite Backup infrastructure for the memory and networking upgrades to accommodate the new global replication platforms and proxies. The maintenance interval is expected to last less than an hour and we do not expect anything out of the ordinary.

For approximately one hour between 2PM and 3PM EST our phone service was knocked out. Due to growth we ordered a block of additional DIDs (direct dial in numbers) and our VoIP IAX provider inadvertently reset all our DIDs to a different proxy server. Unfortunately, we were registering our PBX to the wrong VoIP proxy server and as a result of it, majority of our direct dialin numbers failed to connect. Among major numbers affected were our primary toll free number (877) 546-0316 and our primary local number for international calls (407) 465-6800.

We were able to quickly identify and resolve the problem but suffered approximately an hour of outage without a record of phone calls missed. We are sorry about any inconvenience this may have caused, if you missed us please keep in mind that you can reach most of us via the support portal, email, instant messaging and of course, direct sip dial if you are on VoIP as well we do accept connections at sip.ownwebnow.com  

We are experiencing significant issues with the virtual hosting of Microsoft Windows 2003 servers. We are still collecting data and trying to isolate the issue but some virtual servers simply stop responding to the network requests – either before or after the latest Microsoft security updates. To further complicate issues, virtual machines are crashing upon reboot requiring two reboot cycles.

We are still working on a solution, it appears that after the second reboot the system remains stable and accepts network connections without an issue. Please stand by for update, as of 2:00 PM CMT the entire network appears to be completely operational.