Access to livearchive is currently deprecated. Yesterday we noticed mail submission would randomly drop, so we increased our monitoring. Today, we’re noticing the same behavior, but it’s pointing more to the mailbox databases. We are currently taking multiple databases off at a time and running integrity checks.

We highly apologize for the inconvenience and any partner who has a down server that needs access to livearchive, please open a case with support and we will work out an alternative solution.

We received alerts from our monitoring software that messages on outbound1 were being rejected as relaying denied. We killed the sendmail process and resolved the issue. The bounce backs occurred between 11:20 AM Eastern and 11:35 AM Eastern.

Update 6:44 PM Eastern: The issue reappeared around 6:20 PM and began rejecting messages. We located the root cause, which was a faulty update from an AV vendor. We applied the previous definitions and await an update from the vendor.

Gmail is apparently experiencing technical issues. Several users have reported trouble sending mail to Gmail over the past hour. While we are unable to isolate it at this point we have contacted Gmail and are awaiting response.

The issue does not seem to be widespread, only about 10% of messages sent to Gmail terminate with the 5.x.x series error (permanent failure) and it is being issued by Gmail servers, not ExchangeDefender.

We have alerted Gmail and will update this blog post when they notify us of any problems being resolved on their end.

The livearchive service has been degraded and new mail is not being delivered to the server. We are in the process of truncating log files and optimizing the storage controller.

You may have noticed an unusual rise in the amount of SPAM getting through to select users on your domain. We have isolated dozens of domains against which large scale SPAM attacks have been raised, typically from trusted (reputable) mail infrastructure.

We have been isolating recent outbreak of SPAM that is originating from sources that are typically trusted and managed by larger providers. We have designed new rules, tested them against known sites and our own production domains and we can confirm that we expect the performance as of 2 AM EST to improve significantly. As usual, if you continue to see unusual SPAM levels please use the Outlook 2007 addin and report anything that gets through as the spam@ownwebnow.com alias will be discontinued shortly.

We are in the process of finalizing the upgrades to LiveArchive 3.0. Throughout the week we received complaints from a few partners about delays in mail being delivered to the server and mail being delivered to livearchive. We identified the problem as a bottleneck, which we implemented a temporary solution of la-edge.

During this final upgrade we will have to reboot servers in the livearchive network once before installing the new transport server. Throughout the night we will be flushing out overflow queues from the past 48 hour into LiveArchive mailboxes.

To address several new SPAM patterns and strains that are able to get around the current restrictions (and whitelist implementations) we will be rolling out an update to our AntiSPAM engine at 11:00 EST tonight.

During the 15 minute window around the switch, there may be an unusual amount of SPAM passing through as our databases are flushed, reloaded and propagated through the network.

We have already started receiving support requests regarding missing messages – all caused by not including the latest IP addresses in the IP restriction set.

If you are currently enforcing restrictions, please make sure you have the 206.125.40.0/24 IP address range in your access list:

http://www.ownwebnow.com/blog/2010/03/exchangedefender-grows-again/

Earlier today we launched ExchangeDefender LiveArchive 3, powered by Microsoft Exchange 2010. LiveArchive is a free business continuity solution that is constantly copying all your inbound and outbound messages as they are scanned by ExchangeDefender and in addition to delivering to your mail server or your mail recipients, it delivers to an enterprise deployment of Exchange 2010 in our data centers.

When your servers go down, or production environment is interrupted in any way, LiveArchive can be accessed through any browser or mobile web device and users can have access to up to 1 year worth of email (note: not a compliance or archiving solution)

Passwords

When we went from Exchange 2007 to Exchange 2010 we simply created a new environment and created all the users from ExchangeDefender user databases. You can access your old accounts and your old email at http://livearchive2.exchangedefender.com

Your new accounts are at https://livearchive.exchangedefender.com and they all hold the same username as the one in our ExchangeDefender portal. If you changed your passwords via Outlook Web Access keep in mind that the password will now be synced to our ExchangeDefender user databases and only supported passwords are the ones at https://admin.exchangedefender.com. We will soon restrict the ability to change the password through OWA because users forget their passwords and it creates an enormous amount of support issues.

If you are running into password issues when logging into LiveArchive today please use your ExchangeDefender password, not your OWA password or anything you may have reset it to via OWA. If you need to change the password please do so at https://admin.exchangedefender.com

Have a great weekend and we hope you love Exchange 2010!

We are in the process of cutting over mail flow from our LiveArchive 2 (Exchange 2007) to our LiveArchive 3 (Exchange 2010) cluster. During this transition mail may be delivered to either location, however, users will be able to access their LiveArchive 3 account by 2AM Eastern.

Over the course of the next month mail is scheduled to be imported from the previous LiveArchive cluster.

We’re currently addressing a large scale DDoS attack on our outbound network; We’re slowly clearing out the queues from the attack and the latency has been reduced down to normal levels.

Now that the major launch of ExchangeDefender 5 has been completed and we have our processes together I wanted to resume the helpful NOC posts we did in 2009 that gave you a bit of a heads up about maintenance and network changes that were being made. Here is our agenda for March:

ExchangeDefender upgrade to Exchange 2010

ExchangeDefender LiveArchive has been tremendously popular and we’ve planned some enhancements to the product that will just blow you away. Coming in March, ExchangeDefender LiveArchive will be upgraded to Exchange 2010. Unfortunately due to the volume of mail, a migration will just not be possible (think restoring or mail moving a year worth of your email multiplied by every ExchangeDefender customer – not easy). To alleviate the pain of getting a fresh mailbox, we will continue to run the two ExchangeDefender systems in parallel – old LiveArchive with Exchange 2007 won’t go offline for 30 days after the new one is up and running. Now let me address a concern:

But Vlad, you guaranteed 1 year of archiving free, that’s what we’ve sold to our client and they absolutely need 1 year to be there for their compliance.

This seems to be a common misconception because people focus on numbers and not the language and general spirit of the solution. To be clear, LiveArchive is a business continuity solution that keeps up to a year worth of both inbound and outbound mail in an always on, always available Exchange OWA environment so you can access it at any time and continue working where you left off when your connection was interrupted.

LiveArchive does not guarantee 1 year of archived mail nor is it advertised as an archiving solution – if your clients require that, we do offer a mail archiving solution called ComplianceArchive which is designed for compliance, HIPAA/SOX regulatory archiving and guarantees access to mail for as many years as you wish to archive it for. If you’ve sold your client LiveArchive contact them and tell them that due to migration constraints the new LiveArchive (with new features, see below) will start as a clean mailbox and that they should save any old items they absolutely must have. If they absolutely must have them all, offer to upgrade them to a ComplianceArchive solution.

LiveArchive was designed as a business continuity solution, so that clients that got inconvenienced by server performance, outage, downtime or maintenance could continue working without having to manage a disaster scenario or start from a blank mailbox – typically a day or two worth of email is sufficient to refer back and we offer up to a year because we understand that sometimes people need more and it’s not incredibly expensive for us to provide it.

Now, the beauty of ExchangeDefender 2010 is all in it’s Outlook Web App – no more Web Access -  The new solution is compatible with all browsers which means full mobility and a ton more functionality. We’re also building in rule sets so that outbound mail goes to send items and adding a layer of quality assurance that will address the issues we faced with 2007.

Rollout of Exchange 2010 Hosting & New Control Panels

We will begin offering Exchange 2010 hosting this month as well, with a new quota and a new set of control panels. With the new controls you will be able to manage mail enabled contacts, public folders and distribution groups, so pretty much the same type of control as if you had your own Exchange cluster!

As with all new software, the price is going up but we will keep it at the same level for our ExchangeDefender Service providers. We’ll also be the only company to offer it under your own brand while not competing with you for the business – not directly with our brand and not indirectly through the major distributors. With ExchangeDefender we believe we can deliver the reporting and business intelligence, something that is unparalleled in our industry not to mention integrated with Autotask and ConnectWise.

Other Fun

We’re rolling out new power switches and reboot switches which will make us a lot more green and improve power. There will be several announced reboots / server moves as we move systems around to improve power utilization and cooling. We’ve grown tremendously since ExchangeDefender 5 launched in December 2009 and our products are a hit in the marketplace. ExchangeDefender will be getting a new major data center (delivery point) and we’ll advise of the maintenance as we get closer to the announcement.

We also have some really great EU news but look for that in the newsletter coming around the 2nd or 3rd week of March

Sincerely.

Vlad Mazek, MCSE

CEO, Own Web Now Corp

We have received reports from a few partners that ExchangeDefender users are not receiving mail released from Quarantine. We are investigating into this issue and assure that no mail has been lost. If you experience issues in releasing mail from quarantine please open a support ticket at support.ownwebnow.com including the senders email address , recipients email address, and time stamp.

Update – Noon EST – We have resolved the issue that caused the release mechanism to fail, the problem has been fixed and as of about 11:30 the full functionality has been restored. Now a word from Vlad Mazek, CEO:

As we have widely blogged, we have been migrating from the old systems and old software with the release of ExchangeDefender 5. The final step, which was taken this weekend, involved removal of the old infrastructure at the core of the command center and as is usually the case with complex software, some middleware got in the way.

Just release the messages again and they will flow in. Furthermore, if you’re not familiar with https://admin.exchangedefender.com this would have been a perfect time to check it out, in ExchangeDefender 5 we do offer full access to SPAM so you can view it live without releasing – you can even reply directly from the site!

Again, my apologies about the inconvenience this has caused.

-Vlad

Emai: vlad@ownwebnow.com

Cell: (407) 536-VLAD

Throughout the day the ExchangeDefender outbound grid has been fighting extremely large mail queues and hour long delivery delays. The source has been identified as a DDoS attack and we’ve taken all mesures to remove the mail.

Legitimate mail that hasn’t been delivered will be delivered throughout the next couple of hours.

We highly apologize and we are making modifications to the outbound grid throughtout the next limit to prevent flooding.

We will be holding an extended maintenance window this weekend affecting admin.exchangedefender.com systems:

Sunday, January 31st, 2010

4 AM – 7 AM EST

During this time window access to admin.exchangedefender.com will be intermittent as we undergo a major networking and hardware update to handle the expansion and additional services.

We will start posting updates to this blog during the 4 AM – 7 AM.

The livearchive database for some ExchangeDefender users is starting to show mail routing issues. We’ve disabled this database and temporarily put up a blank database. Over the weekend we will attempt to diagnose the issue with the database and remount the affected database.

Update 12-06-09: After many attempts to restore the database, the decision was made to leave the database dismounted in preparation of LiveArchive 3.0 (Due for release in Feb 2010). The current running database is >6 TB in size and direct repairs would take at least a month, leaving customers without the ability to utilize LiveArchive. All users currently have new mailboxes and we plan to migrate the >6TB database into the new LA 3.0 database.

We’ve received reports from a couple partners that they’ve received an email titled “your mailbox has been deactivated” that has an executable attachment that gets stripped by ExchangeDefender. This seems to be a blind attack from the outside and we’ve already implimented the checks to block these messages from coming through ExchangeDefender.

Just as a reminder, we will never email end users about issues with ExchangeDefender, we only contact our registered partner.

We are in the process of rebooting the livearchive server. Our alerting software showed periods of inaccessibility which we believe will be resolved with a reboot.

Update 8:30 AM Eastern: The livearchive server has been rebooted and is back online. Services are running 100%.

In preparation for the release of ExchangeDefender 5.0 we’ve installed 4 new servers to process outbound mail for ExchangeDefender clients.

This transition was seamless and shouldn’t require any work from our partners, however any clients who are using SPF records will need to add the following IP addresses

p4:65.99.255.234
ip4:65.99.255.237
ip4:65.99.255.238
ip4:65.99.255.239

Over the weekend our replication servers experienced a larger than normal lag in replicating statistical data from our Dallas Colo. This caused quarantine reports to show up as empty as there was no new data replicated when the reports were generated. Reporting is expected to be fully functional by the next scheduled cycle.