Last quarter we launched a slimmed down version of ExchangeDefender to be packaged with Exchange Essentials 2010 and as a standalone product. This product was launched to provide a similar price point and feature set as some less robust Spam & Security solutions out in the channel. However, we were never comfortable with not offering a bundled-in business continuity solution. Enter… ExchangeDefender Essentials Emergency.

Emergency is the business continuity solution that is now bundled in with ExchangeDefender Essentials. It will capture a copy of all incoming email in similar fashion, but only inbound mail with a retention policy of 5 days for all items. This email is accessible via web portal and POP3/IMAP4 (although currently we are limiting the ability on the POP/IMAP to just download messages (to avoid open relay situations). So your clients will be able to continue to do business with the ability to receive, reply to, and create new emails from their real email address during an outage. Remember, that our 7 day spooling/mail bagging system is still in place so a combination of the two should minimize your client’s inconvenience.

The web portal is available at:

https://emergency.exchangedefender.com

Credentials:

Your primary user email address with ExchangeDefender (you cannot log in with an alias address) and your current ExchangeDefender password.

Once you log in all of your email will be available, with your identity pre-configured for use. There is no additional set up required. You can start reading and firing off emails as quickly as you can type.

Setting up Outlook (remember Read/DL message access only currently)

Fill out the information as below:

Your email address and user name are the same as your primary address in ExchangeDefender. The POP3/IMAP4 server is emergency.exchangedefender.com. Everything is on the standard ports for both SSL and non.

Our CEO, Vlad Mazek, will be providing a broader overview on emergency and its feature set when we officially roll it out in the next 2 weeks.

Carlos Lascano
VP Support Services, ExchangeDefender
carlos@ownwebnow.com
(877) 546-0316 x737

One of the most common questions that our support team receives with Hosted Exchange is “How can my client choose which email address I want to send as?” and many partners are shocked to hear the answer “You can’t”.

The above limitation has always existed in Exchange as you can only have one “outgoing” address. Sure, you can create another profile in Outlook and send via SMTP or create a distribution group and add send-as rights..but what do you do if you’re on hosted exchange and you don’t have the freedom to control addresses and configurations at will or find it rather rudimentary?

*Drum roll*

Now you can…

For the first time (as far as I can tell for any hosting provider) partners/clients will have the ability to control their outgoing address in our Hosted Exchange.

Utilizing our API and a brand new API designed by the ExchangeDefender team for direct client Exchange interaction we now can extend the ability for clients to control their outgoing address on the fly.

The current beta product is a windows based application and will soon be converted to an Outlook 2010 plugin.

If you or your clients are interested in test driving this software please reach out to me directly: travis@ownwebnow.com.

Note:  This only applies to rockerduck.

Travis Sheldon
VP, Network Operations, ExchangeDefender
(877) 546-0316 x757
travis@ownwebnow.com

You may be saying “Wait, this sounds impossible” or “It is too good to be true.” That would be because it really is.

It is not possible to honestly position any one email product so that absolutely everyone will see real value in it – there are just too many varying sets of needs and requirements out there. But you can quickly and easily get a few pieces of information that you can use to show your clients how they can take advantage of the wealth of value within Hosted Exchange 2010 + SharePoint 2010 from ExchangeDefender.

Your first task when you begin the sales conversation with any new client or prospect is to find out what Hosted Exchange 2010 + SharePoint 2010 can do for them. How do you do that? By focusing NOT on the hosted exchange product or service, but rather on the client himself (or herself) and asking them the right questions, the right way.

In this questioning process you will find out what email service they are currently using, what issues they have with that service at that time, and how those issues are affecting them. The implications, or how their issues with their email service affect them, are the real meat that you are looking for.

What can Hosted Exchange 2010 + SharePoint 2010 from ExchangeDefender do for them? Chances are good that once you understand the issues and how they are affecting the client’s business, you can very easily illustrate how Hosted Exchange 2010 + SharePoint 2010 will alleviate those affects by eliminating the issues.

For example, if you hear from a small business client running someone else’s web-based email that the limited support is extending downtime and costing the client business, then you know that a brief mention of the quick and effective 24×7 support available through ExchangeDefender might be highly enlightening to them, and may serve to enhance your value proposition significantly.

Remember, positioning any product is all about building value proposition and painting a picture. Our CEO, Vlad Mazek, calls it “Selling the Dream” – help the client see exactly how much better their lives and business will be with Hosted Exchange 2010 + SharePoint 2010 from ExchangeDefender and closing the sale becomes the natural, easy result of the conversation.

Do you find this information useful?

If you would like a lot more in-depth discussion about the cloud and how it affects you and your clients, visit Looks Cloudy http://www.lookscloudy.com where I blog daily about the adoption of the cloud in SMB, conduct live webcasts and podcasts with industry leaders, and more.

Kate Hunt
VP Community Development, ExchangeDefender
kate@ownwebnow.com
(877) 546-0316 x777

Hosted Exchange 2010 + SharePoint 2010 from ExchangeDefender can address a lot of different kinds of problems for a lot of different kinds of businesses. It can help many of our clients in a multitude of ways. Hosted Exchange 2010 + SharePoint 2010 from ExchangeDefender minimizes the upfront costs for a business to take advantage of stable, secure web communication and collaboration. Because it’s hosted in our cloud, it’s accessible from anywhere, 24 hours a day 7 days a week, on any device. We include everything that a business needs to leverage the power of Microsoft Exchange for their growing business for a low monthly fee.

Here are three common business challenges and how Hosted Exchange 2010 + SharePoint 2010 from ExchangeDefender solves them:

Losing time, wasting time, or not getting enough done as a group

For growing businesses, profitability is all about efficiency – doing more with fewer resources, less money, less time. It is mission-critical to make sure that employees are actually working, and can be really important to be able to see who is free to take on a job at the last minute, at any time.

Hosted Exchange 2010 + SharePoint 2010 from ExchangeDefender allows your clients and all of their employees to share calendars and tasks easily across the web. They’ll be able to tell who’s busy and who’s free right inside Outlook, schedule appointments on other users’ calendars, and save time by avoiding all the complexity of phone calls and emails for scheduling and coordination.

Lost or delayed orders from email server crashes

Many businesses today use email to track orders, orders from their suppliers and vendors, and also orders from their clients. When email is down, it can take a severe and expensive toll on the business.

With Hosted Exchange 2010 + SharePoint 2010 from ExchangeDefender, data is stored on our globally-redundant infrastructure and accessible via Outlook Web Apps, Outlook desktop client, and mobile devices. Hosted Exchange 2010 + SharePoint 2010 from ExchangeDefender is protected by the world-class ExchangeDefender security suite, so in addition to being reliable, messages will be free of malicious code and safe for consumption anytime, anywhere, and also comes with the LiveArchive service built right in. LiveArchive is the ultimate redundancy feature that enables real-time access to one year’s worth of messages via Outlook Web Access anytime, anywhere in case of any kind of hardware failure or outage.

Losing money by being chained to the office park

So many businesses today are beginning to recognize the opportunity. There are many efficiencies and benefits to deploying a mobile workforce. Smartphones, tablets, telecommuting, and the like are becoming more accessible, less expensive and generally more viable options for all businesses, especially growing ones.

Hosted Exchange 2010 + SharePoint 2010 from ExchangeDefender supports automatic wireless synchronization with your Android, iOS, Windows Phone 7 device. BlackBerry support is available for a small additional fee, as well. Whichever mobile platform the client is on, contacts, calendars, and messages are kept up to date at all times.

…and another thing:

Speaking of losing money, perhaps the number one limitation faced by growing businesses is being tight on capital. Hosted Exchange is available for a flat monthly fee per user without the big up-front cost of servers and software licenses that come with traditional applications. It’s fully compatible with existing supported Microsoft Office desktop applications, and fully supported 24/7.

Do you find this information useful?

If you’d like a lot more in-depth discussion about the cloud and how it affects you and your clients, visit Looks Cloudy http://www.lookscloudy.com where I blog daily about the adoption of the cloud in SMB, conduct live webcasts and podcasts with industry leaders and more.

Kate Hunt
VP Community Development, ExchangeDefender
kate@ownwebnow.com
(877) 546-0316 x777

The past two weeks have been rather stressful and unpleasant for our partners and your end users on DEWEY due to the recent outage that left 17% of the users without full access to their entire mailbox for two weeks. Not only has this been our longest outage with any service, but it also has been one of the most “bumpy road” recoveries we’ve ever experienced. Our partners and their end users could compare almost every step of the recovery to pulling teeth; every step of the way, we let them down.

We’re extremely sorry…this was (and still is) extremely unfortunate, but we have learned a lot from this experience.

Here’s a quick break down of the experience:

10/31/11 – Users on two databases both lost connection to Exchange.

11/01/11 – Users are switched to dial tone recovery mode. Users with mobile phones lose contacts and calendar events on next sync.

11/06/11 – Users on DB2 regain access to old mail, but lose access to any new mail, contacts, or calendar entries since the outage began.

11/07/11 – Mailbox data for users on DB2 is fully restored; new and old data merged.

11/17/11 – Users on DB3 regain access to old mail, but lose access to any new mail, contacts, or calendar entries since the outage began.

11/19/11 – Mailbox data for users on DB3 is fully restored; new and old data merged.

In speaking to partners along every step of the way we heard every issue experienced by end users with the biggest issue being the dial tone recovery. During dial tone recovery any users with ActiveSync based connections will lose all Exchange contacts and calendar items on next sync after dial tone is activated.

When you break down our responsibilities and duties to our clients, at the very minimum, we need to provide a live running service as quickly as possible. Technology and software unfortunately have issues and can break, but as long as we can minimalize the direct impact faced by end users we can generally get through issues without upsetting a lot of partners. By erasing all cached access by mobile phones with the temporary mailbox, we put a lot of stress on our partners.

We are considering a disaster recovery policy control for partners..

The policy controls will give partners control of how we treat each mailbox during an outage. In giving partners the ability to control how we respond, this will greatly improve the overall experience during an outage and will allow the partner to provide direct expectations to their clients.

For instance, say partner ABC123 Computers has a client, Big Electric Company, with 10 mailboxes.

The partner wants to mark 3 users as “Do not activate dial tone” as these users are mobile and depend on their contacts and calendars and they cannot afford to lose them once dial tone is activated.

The partner then marks the CEO and CFO as “Do not reimport data after outage” as the partner plans to reimport cached data because the CFO and CEO cannot work on dial tone mail alone.

By allowing the partner to directly control our recovery process for mailboxes, the partner will then be able to set direct SLA expectations for their end users during the outage.

Plans for the future…

In the event that we experience another catastrophic failure (anticipated 6+ hours of downtime), we will wait at least one hour before activating dial tone recovery for mailboxes that are not opted out.

During the first hour, we will reach out to all partners via telephone whose mailboxes are not opted out from dial tone recovery to make them aware of the expected experience for end users. If the partner wishes to not activate dial tone recovery we will activate the mailbox option in service manager to opt out of dial tone recovery. Additionally, partners could ask (via support ticket) that the dial tone activation to be postponed for a few hours later if the partner wants to advise the end user to disable ActiveSync.

Once the original (or backup) database is back online we will once again reach out to all partners who have not opted out of mailbox data restore to let them know of the expected experience for end users. If the partner wishes to opt the mailbox out of data restore then the mailbox will remain on the dial tone database until the recovery of data for all users on the affected database is completed.

If you think the outage policy control would be a beneficial add on to our Hosted Exchange service, please let me know travis@exchangedefender.com.

Travis Sheldon
VP Network Operations, ExchangeDefender
(877) 546-0316 x757
travis@ownwebnow.com

Hosted email can be very simple – pay one flat price per month per mailbox for a set of great services that meets all the client’s needs. Yet some providers choose to make it very complicated – just determining what the per-mailbox cost will be can seem like solving an algebra problem. Hidden add-ons and fees abound, and add up very quickly to costs much higher than anticipated. When anything changes, like storage needs or mailboxes or additional services, account changes can be complicated, may affect the price dramatically (which wreaks havoc on your profitability), and can be difficult to manage.

At ExchangeDefender, our Hosted Exchange 2010 + SharePoint 2010 offering makes it very simple. For one flat, all-inclusive rate per mailbox per month, you can offer your clients a hosted messaging solution that is flexible enough for their varying needs today and tomorrow, and robust enough to enable you to turn a profit on every mailbox every month.

“Under promise and over deliver”

Let’s say your client, like so many small businesses, indicates in the sales conversations that they “just need a basic email solution” that will work and keep them safe and secure. Well what if you could offer that to them, at a great price, with all the features that they could ever want to “add on” later built right in and ready to go when that time comes?

ExchangeDefender Hosted Exchange 2010 + SharePoint 2010 solution comes with full ExchangeDefender protection, an Outlook desktop license, Live Archive service, web filtering, web file sharing, and even more great features built right in. If your clients don’t need or want any of them, you can simply disable them. You can deliver that fully-customized functionality with just the click of a button in your management portal. There’s no migration, no additional provisioning or payments, no updates. It’s just that simple, for you and your clients.

We offer ExchangeDefender Hosted Exchange 2010 + SharePoint 2010 to our partners for $10 per mailbox per month, with additional discounts available based on volume. You get to mark this up and charge your client at your discretion. We typically see the service being resold for $15-20 per mailbox per month, providing a nice healthy margin to the MSP partner. And the price includes everything, with no fine print or hidden catches. $15 a month for a full-scale, customizable Microsoft Exchange email service sounds very appealing.

It’s easy to manage, so you’ll spend a little less time managing each client’s changing email and messaging needs, which adds up to big impact on the bottom line. Remember that when the fee is fixed, the goal is to minimize your internal costs for managing and delivering each service to each account, to be most profitable.

Do you find this information useful?

If you’d like a lot more in-depth discussion about the cloud and how it affects you and your clients, visit Looks Cloudy http://www.lookscloudy.com where I blog daily about the adoption of the cloud in SMB, conduct live webcasts and podcasts with industry leaders and more.

Kate Hunt
VP Community Development, ExchangeDefender
kate@ownwebnow.com
(877) 546-0316 x777

A few weeks ago (10/31/11) our DEWEY cluster suffered a major outage at the result of old conflicting RAID policies that led to the dirty dismount of 50% of the DEWEY databases. Immediately, our team started the recovery process, which unfortunately took longer than ever expected (More details @ https://www.exchangedefender.com/noc) and prompted our side to start developing a new solution for DEWEY users to restore and maintain the expected service availability and reliability.

Lets be forthright…DEWEY has been somewhat of a nuisance not only to our staff, but also to our partners and your users.

Overall, Exchange 2007 was a drastic leap and improvement from Exchange 2003, but still lacked many features for service providers or any rapidly expanding company such as the ability to properly scale the solution and implement additional failovers after initial deployment. In comparison, I think of Exchange 2007 as Windows Vista – it brought a lot of new features and easier management from 2003, but after a year of management it feels like a bad tease.

Over a year ago, we decided to immediately start offering Exchange 2010 as soon as it RTM because we felt we hit the ceiling in terms of scalability with Exchange 2007. Sure, we could add more CAS servers and MBOX servers, but that would eventually end up with partners having multiple configurations to keep track of for clients in the same company. Taking everything into account Exchange 2007 was no longer the road we traveled down for new deployments.

Going back to DEWEY, we started experiencing more frequent issues in terms of reliability over the past six months. Between services failing to respond in a timely manner or even worse, not responding at all we knew we had to provide a solution before our partners clients threw up their hands to say “I’m done!”.

Upgrade, Upgrade, Upgrade

In order to provide a solution we must keep a few essential components in mind:

1. The upgrade must be seamless with no mail loss and no major service interruption.

2. The upgrade must be automated. We can’t expect partners to import/export data or reconfigure Outlook.

3. The upgrade must be scheduled so partners are kept in the loop on the migration progress.

4. The upgrade must be flexible for partners who want to slowly migrate or for partners who want to immediately move.

Unfortunately, Exchange 2007 does not provide the ability to seamlessly move users directly to any remote Exchange 2010 cluster (i.e. Rockerduck), but we can upgrade to a local 2010 cluster, then move the client to ROCKERDUCK.

The Solutions is Here…

In the past few weeks we have deployed and successfully tested a 40 user migration from 2007 DEWEY to 2010 ROCKERDUCK with very minimal effort on the partner’s end. Below is a rough overview of the process which was designed for a 2 week migration for clients:

1. Partner adjusts autodiscover CNAME record to NEWDEWEY (2010).

2. Move requests get created from 2007 DEWEY to 2010 DEWEY (Week 1, Monday).

3. Users on DEWEY will remain on 2007 throughout the work week.

4. Move requests will be finalized on Friday evening. Users will now be serviced on 2010 DEWEY (Week 1, Friday).

a. Users with BES devices will remain on 2007 until the final move to ROCKERDUCK.

5. Users come in on Monday, restart Outlook.

6. Move requests to ROCKERDUCK are created (Week 2, Wednesday).

7. Partner adjusts autodiscover record to point to ROCKERDUCK (Week 2, Friday at close of business).

8. Move requests will be finalized Saturday evening (Week 2, Saturday).

9. User comes in on Monday, selects “Repair account” on ‘Email Accounts’ list. User is now serviced on ROCKERDUCK.

10. Users re-setup ActiveSync and any Blackberry Enterprise Server accounts.

With the above outlined process users should not see any service interruption nor will users have to recreate their Outlook profiles.

Travis Sheldon
VP Network Operations, ExchangeDefender
(877) 546-0316 x757
travis@ownwebnow.com

On a random basis, one of the most pressing issues faced by our customers was mail delivery delays. These random delays happen to only about 12% of the nodes, however, due to the sheer volume of mail processed by our inbound network that 12% would inevitably cause our staff and customers quite an inconvenience.

As more and more companies begin to depend on email for the main source of communication for their business, mail delivery time becomes a major factor when partnering with a hosting vendor. Because of the critical need for instant delivery, we had to quickly overcome our growth and produce an immediate solution.

The Unforeseen Weakness in Round Robin

After the ATS power outage earlier this quarter we were forced to reevaluate or solutions across the board and make drastic improvements. One of the areas that desperately needed a redesign was our inbound mail load balancer.

In the early days of ExchangeDefender we utilized a round robin based load balancer. In short how it works is the MX records for ExchangeDefender clients are pointed to both our Dallas datacenter and our Los Angeles data center. After the SMTP connection hit either data center, the connection was then forwarded off to any random inbound node in the virtual server list.

Until earlier this year, the round robin design worked quite well, however, as the number of messages being processed grew, so did the delivery delays. We started to notice that the load balancer that was able to previously balance the connections was no longer balancing at all. Day after day, we saw some inbound servers having upwards of 200 concurrent connections at a time.  More than half of the other inbound nodes in the respective data center had no connections at all.

The biggest issue preventing the round robin configuration from working was the randomized assignment of which data center would be used and which inbound server that would receive the connection.  To begin to tackle the issue, we had to re-evaluate the entire load balancing solution because you can never properly balance a round robin based connection. We switched our load balancers to use a weighted least connection based routing scheme.  Upon activation it seemed to balance connections a bit better than the round robin connection.  Nearly an hour after activation however, we saw a large queue size being placed on a few inbound nodes.

Brand New Logic

To completely resolve the issue, we had to introduce additional logic to the load balancer. The recurring issue we faced was the basic nature of SMTP. An SMTP connection for one “message” being transmitted could equal four concurrent open connections. Therefore, naturally, the connection count cannot be relied upon for load balancing. We then decided to leverage our queue reporting service which reports the number of queued messages and open conversations with unique IPs. Finally, we created a PHP script that runs on the load balancer and splits itself to check the connection counts across all nodes every 10 seconds. We used a very simplistic formula for load balancing:

If(($numActiveServersInSite >= ($numServersInSite / 2)) && ($nodeConnections >= ($nodeAverageConnectionCount + 50)) && ($nodeConnections <= $highThreshold)){Shutoff new connections}

In non-code terms, we now calculate the number of servers in each site (DC). If there are more than half of the servers offline in a site, the load balancer will no longer shut off new node connections. If the current node connection count is either greater than the high threshold or has 50 active connections above the average connection count of servers in the site then it will shut off the node.

The End Result

After implementing the new load balancer algorithm we saw drastic improvements of balanced connections and delivery times for inbound messages. The most notable improvement took place on September 10th where we processed 2.8 million messages and we saw minuscule delays across the board and received zero complaints about deliver delays.

Travis Sheldon
VP Network Operations, ExchangeDefender
(877) 546-0316 x757
travis@ownwebnow.com

Have you considered adding hosted Exchange or other cloud messaging solutions to your portfolio? If not, you’re already behind Microsoft, Google, IBM and many others. Even if you have considered it, the marketplace is getting flooded with cloud marketing and you need to be savvy to get your solution front and center and not get displaced by he competition. Here is how! 

Hosted Exchange 2010 + SharePoint 2010 and Exchange 2010 Essentials:  while they’re only two simple, reliable solutions, they pack a powerful punch that solves a wide variety of small business messaging and collaboration challenges.  When we surveyed our partners, 90% of them told us that Hosted Exchange is their single best “foot in the door” or relationship builder product.  This is because small businesses are becoming more and more dependent on their messaging technology, while the cost of deploying and managing it continues to skyrocket.  They need affordable, reliable messaging technology to support their unique businesses – and Hosted Exchange + SharePoint and Exchange 2010 Essentials from ExchangeDefender offers small businesses of all kinds an attractive combination of rich features, affordability, security and flexibility. 
 

Differentiation

Our small business clients certainly have options when it comes to cloud-based messaging solutions – Microsoft Office 365 and Google Apps for Business are two major alternatives with which many small business clients may be familiar.  Hosted Exchange 2010 + SharePoint 2010 and Exchange 2010 Essentials from ExchangeDefender are fundamentally different, offering users the familiar Microsoft Outlook user experience at an unbeatable value with the flexibility to deploy just the right tools that the business needs. 

Some SMB clients don’t want or need a full-featured messaging and collaboration solution and want to minimize their costs of maintaining basic functionality.  While Google Apps for Business offers an attractive price point with “pared down” messaging features, its unfamiliar interface may turn off folks who are comfortable with the Microsoft user experience.  Exchange 2010 Essentials delivers the best of both worlds – minimal complexity using Outlook Web Access protected by ExchangeDefender Essentials along with the ever-popular Public Folders and Distribution Groups at a competitive price point, just $6 per month per mailbox.  Clients can even integrate with their existing copies of Outlook to get that fat client experience they know and love.   

Feature/Benefit Discussion

Some SMB clients will require more advanced features like message encryption or archiving for regulatory compliance, and may find the messaging features included in Microsoft Office 365 and Google Apps for Business lacking – security, encryption, and archiving aren’t included in Google Apps for Business, and while Microsoft Forefront protects Office 365 email users, archiving, encryption, and e-discovery aren’t included.  Hosted Exchange 2010 + SharePoint 2010 delivers the full Microsoft Outlook experience with SharePoint’s powerful online portal and groupware functionality, including ExchangeDefender protection and our Live Archive service.  Integrate with Blackberry Enterprise Server, split domains, brand the portal experience and leverage the “tried and true” Public Folders feature if desired, as well, all for only $10 per month per mailbox. 

Whatever your clients’ needs are, you can get your foot in the door and start building a strong relationship (or deepen an existing one) with the powerful combination of Hosted Exchange 2010 + SharePoint 2010 and Exchange 2010 Essentials from ExchangeDefender.  Now you can offer your clients an attractive combination of rich features, affordability, security and flexibility on our globally-redundant infrastructure that even most large enterprises cannot build in-house. 

Do you find this information useful?

If you’d like a lot more in-depth discussion about the cloud and how it impacts your clients join at Looks Cloudy http://www.lookscloudy.com where I blog daily about the adoption of the cloud in SMB along with numerous webcasts, podcast with industry leaders and more.

Sincerely,
Kate Hunt
VP Community Development, ExchangeDefender
kate@ownwebnow.com
(877) 546-0316 x777

As you may be aware, we had our first network-wide outage in our primary data center (Dallas) back in August and our redundancy plan did not deliver on the promise that we should expect. Our CEO went over the details of the outage and during his webinar, we made a promise to revamp our network and infrastructure to prevent the impact this had on our partners moving forward. Once all services were restored, we began playing the outage backwards to figure exactly what went wrong and how we can eliminate the possibility of it ever happening again.

The items that floated to the top were LiveArchive’s failure and an unforeseen level of reliance on Dallas by our Los Angeles ExchangeDefender network. Our Exchange Architect and lead engineer, Travis Sheldon, outlines the changes made to our LiveArchive structure to ensure that if Dallas goes lights out, LiveArchive can keep on to ticking without missing a beat.

That leaves changes to our mail processing. Once the outage happened, our ExchangeDefender NOC team was overseeing the mail flow going to Los Angeles, however, the failure point was the processing speed. The processing speed delay basically blew out all of our disaster recovery metrics that showed us that LA could handle the load on its own. The team’s task quickly became finding the delay within the process. In this case, our transparency became our worst enemy. The fact that we log everything to make it accessible to our partners for their clients was causing the delay. Our Los Angeles nodes were attempting to write data back to an unresponsive database. Think about that for a second, every step of the way for every email we process, was now attempting to update data to an unresponsive network. Needless to say this delay in processing caused all of our queues to grow explosively, thus causing massive delays in delivery.

The other major part of the outage was our outbound network. At the time, our multihomed smart host cluster was at the time housed primarily in Dallas, with standby nodes in Los Angeles. Here’s the problem, since these nodes weren’t in production they did not have active access tables, which basically rendered them useless. So now we had our trail of blood and tears of how and why we failed, but we also had a blue print on how to ensure history didn’t repeat itself. Here is what LiveArchive side of LA looks like now:

So in addition to the new LiveArchive Exchange 2010 cluster in Los Angeles, here’s what we did:

1. We increased the hardware and bandwidth capacity by 80% in Los Angeles. This brought it on par with our Dallas network, which has in the past and with ease handled the load during peak on its own without delays. In addition, we also increased our capacity in Dallas an additional 20%, this is accommodate future growth without having to experience any processing delays.
2.  We deployed master to master replication of all of our core processing databases. So now our ExchangeDefender Los Angeles nodes do not rely on any Dallas resources for mail processing. This means two big gaps were closed, if there were a Dallas outage, processing speed would not suffer and that our logging would not need to be sacrificed in the name of processing speed.
3. We doubled our outbound capacity in Los Angeles and they’re live servers instead of standby. This way if an outage occurs, all routing/access/archiving/encrypting rules are already up to date and ready to go. As an added benefit this expansion increased our outbound processing speed and capacity by 80%.

The lessons we learned from this event will only allow us to provide you with better and faster service, so you can in turn deliver the same to your clients. We’ve already successfully stress tested our new infrastructure on multiple occasions with great success and we’re confident about the future of our solutions.

I have worked with many of our partners through the years and, if you don’t mind, I wanted to share some resources that I find most of our partners are not aware of. As the person that oversees all of our support services I can tell you that frustration is a part of the game and we can extend a much better service when you’re plugged into everything that the support side of ExchangeDefender as an organization offers. Here are some tips:

1. Make sure all of your employees are reading our NOC site. http://www.exchangedefender.com/noc While the NOC alerts sync up with Twitter, Facebook and our support portals, our NOC site provides a lot of useful information about how to explain the problems that happen. Our clients are aware things will go down from time to time so keeping them in the loop is critical.

2. Make sure all your employees are in the support portal and that the information is correct. Have you ever had the frustrating experience of opening a ticket only to be asked for more information? We find that most tickets that aren’t addressed by us immediately (and our average response time is a topic for a different blog) can drag on and a quick call can often resolve it. We’re in support, we won’t call you to sell you stuff. But if we have a number to call we can figure things out very rapidly.

3. Rely on the portal. https://support.ownwebnow.com is where we live. All of our communications, alerts, NOC, service monitoring and staff have this open whenever they are at their desk. Monitors around the office show how many issues are being worked on, what everyone is up to, which changes are being made. While we love to talk to our partners we have to take your security into account so make sure that everything goes through there. Yes, we do escalate stuff to Vlad from time to time.

4. Most importantly: Save the info below.

I have been with Own Web Now for years and I often like to tell people that I’ve known Vlad Mazek before he knew how to speak English. If there is anything on the support side that I can help with please feel free to contact me. So much for the introductions, look forward to with you all.

Carlos Lascano
VP Support Services, ExchangeDefender
carlos@ownwebnow.com
(877) 546-0316 x737