This weekend we will be revealing the first set of patches that have plagued nearly all of our products. Additionally, we are migrating away from several technologies that have, frankly, failed us and failed our partners for the level of quality you should expect from Own Web Now. We are not willing to point any fingers, you and your customers trust us to deliver rock solid solutions and at the end of the day it’s our decision to make the right choices.

Unfortunately, the reality is that we haven’t made some right choices in the past two quarters and in April we will be moving to address those issues, in all products across services.

There will be extended maintenance hours each weekend outside of business hours as we move to address the many problems noted in the NOC blog over the past few months.

As always, I appreciate your business and I speak to partners every day that love what we’re able to do. That is what drives us and my team strives to deliver more every single day. That is what we are good at – competing. We have dedicated 2010 to raising Own Web Now to the next level, so I hope you pardon the dust throughout April as we hold extended maintenance hours to bring a level of consistency.

I promise you that each and every step taken, to address some of the issues the few of you have been very vocal about, will be fully communicated through this blog.

Maintenance period completed, services back at 100%.

The quick answer is: No, we have not diminished the importance of the NOC blog.

The long form answer is that our service levels have improved significantly and with the improved staff, training, redundancy and several long-term problems addressed, we have not had many network events that warranted posts. The staff is busy working on the new products we will be launching this year and with the more aggressive growth in 2009 we have been far more aggressive in adding resources before the demand, not after.

As always, our service has improved thanks to your feedback and your continued business. We appreciate it all very much!

We have had an exciting 2008 with all the ExchangeDefender and Offsite Backup work. On behalf of my team I am sure you will appreciate that we hope for a far less exciting 2009

Ongoing Issues

None

Weeks Agenda

Exchange Hosting – We are rolling out a new Exchange 2007 grid, huey.exchangedefender.com today. This is primarily to address the growth in our Exchange Hosting operation and eliminate any growing pains down the road.

ExchangeDefender AntiSpam – As announced on our official blog, the new ExchangeDefender engine is in play. We will be keeping an eye on this for any unusual behavior over the next few days.

Web Hosting – Migration to new hardware continues. We expect to complete web server migrations this week with SQL scheduled for the weekend.

Week in Review

Last week included extensive performance enhancements to our Enterprise Storage network and our ExchangeDefender network. Our ExchangeDefender network had a few network topology scenarios isolated that would create random mail delivery delays, something we have now addressed. Both our XD and ES networks struggled with a network hardware upgrade that really did not live up to the expectations. Unfortunately, when something doesn’t live up to expectations in an enterprise production network, discontinuing the use of that hardware is never a simple unplug – it is a slow migration and waiting for massive amounts of data to be replicated onto the replacement systems.

Great news is, our ExchangeDefender network and our Enterprise Storage network are now rock solid and we feel the additions and modifications over the past week will allow us to easily scale up the additional 30% we are predicting for the first quarter of 2009. Seeing what we’ve done with the “projections” in the past we are well under way to announcing the new Los Angeles data center currently being built out to support our ExchangeDefender network alone.

We wish you a Merry Migration

We are currently working on a massive upgrade to our virtual hosting infrastructure, the biggest one we’ve had since 2003. While this work is scheduled to last close to a month our first objective is to get the new hardware in place and move all the services to the new systems and establish a working baseline before performing the management software update towards the end of January.

Starting this week you can expect slight downtime (less than a minute per vserver) as the data rsync’s between the old and the new systems.

We are light on entertainment today but very big on caffeine-induced energy because the weekend network infrastructure upgrades were quite uneventful. We have swapped out a good amount of aging hardware and hope that the upgrades to the network can keep up with the growth cycle. Speaking of which, we are officially invading Canada this week with native Exchange 2007, Offsite Backups, Web Hosting, SharePoint and then some. Over the past year we have grown significantly in Canada and the demand for localized services has been great.

Looking Back:

Issues with RBLs – We constantly monitor our outbound mail flow and the RBL submissions and notifications. Because we own our IP address space we are notified whenever an abuse notification is filed against our IP space and we move to quickly deactivate the offending client. In instances when the IP address is blacklisted without our knowledge, or by a list that is not reputable, we follow the usual procedure of requesting delisting and re-routing the mail out through an IP that is not blacklisted.

What should you do? First, if you received a notification that the IP address was blocked you need to contact the recipients email server administrator. ExchangeDefender will make the best effort to contact the postmaster@ of the sites we have issues getting mail through but due to the volume and backlog our action will never be as fast as yours. Second, you should attempt to resend the message. Most of the RBL complaints we field at ExchangeDefender are not legitimate but just random rejections that overloaded servers make. Third, if this is a frequent / important client you should create a direct SMTP connector to their mail server and establish a trust with the organization. Finally, if you participate in any kind of broadcast mail, distribution group, personal mail lists or in any way send identical messages to multiple people we will not be able to assist you. Although you may not think it is SPAM that you are sending you have to understand that most antispam products use what is called Heuristics and bayesian analysis that tracks identical messages being relayed – computers are blind to your relationships with your customers, the double opt-in or even joke messages. Some of the largest service providers even keep a statistical model of messages and senders that have most of their mail moved to the Junk folder. So if you start to see and hear notifications that your mail is suddenly not getting to the recipients please consider outsourcing your mass/bulk mailing services to another email address or preferably any company. Important:  Using ExchangeDefender for any mass mailing activity is a violation of AUP and outbound services can be suspended if the client gets caught. At the end of the day it really just comes down to courtesy: You can’t pay people to protect you from the junk mail you don’t want to read while you are directly contributing to the problem.

Issues with slipping SureSPAM – We have seen a significant pickup in these. Spammers, having been taken down in every imaginable way, are now exploiting the last bit of user stupidity – mirror whitelist. Here is how it works: Spammer will forge a message with your email address and send it to you. Your message will naturally end up in a SPAM bucket but far too many people have gotten used to clicking Trust Sender or WhiteList and have now created what is called a mirror whitelist – from me to me. When future SPAM messages come from this forged sender (you) they will automatically pass on through because no SPAM checks will be executed against a message placed in the whitelist. We often even find email administrators adding their own domains to whitelists on the global settings so please if you see [SURESPAM] or [SPAM] make sure you clean up your whitelist.

Looking Forward: Big launch week, keep an eye on http://www.ownwebnow.com/blog

On behalf of the NOC team thank you for your feedback. Many of you ask why we only publish the negative and service-affecting issues. That is what this blog is all about, things that are down or are going down that we are trying to fix because they are supposed to work, always. But if we must brag, the upgrades and hotfixes on our Exchange 2007 network have been incredible, all systems have been rock solid for a while now without a second of outage. The recent update to ExchangeDefender XD engine are filtering out more SPAM than we’ve ever been able to with the record new lows in false positives. So much for the good news

Items we are working on:

Upgrades: Thanksgiving weekend is when we make massive hardware upgrades to our shared services infrastructure. As devices age we routinely replace them (even if they are in perfect functional order) so Thursday – Sunday will involve a lot of new parts replacing the old parts. There should be no system outage until we replace the load balancers and that switch will be announced separately in advance.

Offsite Backup Scaling: Our Enterprise Storage Network is growing again, there have been some sporadic issues that we are still trying to address with EMC.  Typical issues involve no email reports being generated because the backup job hangs at the last process (report email) and so far we’ve been able to isolate it to a write timeout.

Items that we are concerned about:

Exchange 2007 SP 1 Update Rollup 5: We will not be immediately rolling out the backup in production even after it clears our labs. We will be using the package on our intranet in production and the two volunteer sites before rolling it out to the remainder of the Exchange 2007 network.

See you in December!

<<< 521-65.99.255.236 blocked by sbc:blacklist.mailrelay.att.net.

<<< 521 DNSRBL: Blocked for abuse. See http://att.net/blocks

554 5.0.0 Service unavailable

We have put in place a workaround and are working with AT&T to resolve the issue. You should not continue to see this problem. However, the issue is still open.

Extended maintenance windows over the weekend have been completed successfully and have addressed numerous growing pains and performance issues. Biggest changes were applied to our Exchange 2007 network and all the systems are now performing remarkably well. We would like to thank Microsoft CSS Exchange team for all the assistance they have rendered over the past two weeks in isolating creeping and unusual issues with the product that we have never encountered before.

Issues we are currently working on:

ExchangeDefender Reports Replication – At times the replication between our master maillog servers and slave reporting servers tends to lag. Sometimes the replication lags beyond our tolerance and appears to nearly stall. We have not yet isolated the issue that is casing this problem but are monitoring it very closely and adjusting replication points.

Offsite Backup Scaling – We are preparing our offsite backup network for another product addition to the matrix. You may experience slight availability issues as the management consoles reload while changes are taking place. This process will not interfere with the execution of offsite backups because the backup agent software automatically reloads.

ExchangeDefender SPAM Load – New ExchangeDefender antispam engine went online last week and the detection has been improved dramatically. We still haven’t kicked it up all the way but expect to by Wednesday. We have had several complaints about the level of SPAM and in our investigation have found that almost all of them are related to end user stupidity – whitelisting null senders <> and whitelisting their own domain or email address. When you whitelist any address you suppress ALL SPAM detection mechanisms, so when you whitelist your own address you open yourself up to a ton of SPAM. We have a process that we are preparing that will automatically remove null sender and mirror trusts because those should never be programmed in.

Lastly, an apology to our Canadian clients. Last week we let the cat out of the bag about our Canadian offsite backup product. This site is not intended to serve as an announcement board for new products, we just want to keep you informed about what our group is working on. From time to time what we are working on may not match the business side of Own Web Now so if you have a product or business related question we are not the most authorative source to provide an answer.

Additional 600 nodes are planned in our global expansion leading up to ExchangeDefender 4.0 launch. We are also looking at additional data centers on both coasts at the moment scheduled to go live this fall.

Update: 2:24 PM EST: We are happy to report that all the nodes have now converged in the scanning network and the SPAM filtering is back at its usual levels (and to be tightened up even further later tonight). You may have seen an increase in SPAM over the past few hours while the nodes were joining the network and accepting new programming but you should be seeing far less SPAM going forward.

If you are experiencing an issue, please open a support request and include a traceroute to the service you are trying to reach (ex: 65.99.255.50).

We will update this ticket as soon as we have further information. 

Update (9:05 AM EST, 14:05 GMT): We believe the network issue some of our customers have experienced has been resolved. Particularly affected were some of our UK customers (not BT) and local customers with Level 3 connectivity. Although we see the traffic back up at usual levels, it may take about an hour or so until all the mail catches up and gets delivered.

Please remember that the primary way to check for SPAM is through the ExchangeDefender portal at https://admin.exchangedefender.com and for the customers that need a reliable, realtime reminder of the SPAM that is being kept from their mailbox we recommend the SPAM Monitor software designed to sit on the desktop and show alerts. We still consider email reports to be an important part of our offering but due to the their nature (spam digests of spam messages) we urge you to consider the more reliable, realtime alternatives.

As a result, our phone systems may not be the most reliable way to reach us today, January 2nd, 2008. If you have an urgent issue, or even an issue that you would like human followup on, please post a support request at https://support.ownwebnow.com.

If you need immediate assistance please remember that we only provide sales and support via our portal at https://support.ownwebnow.com

At peak, our network was 81% saturated on Friday, AM which is far above 25-35% it peaks at during the week. The current network utilization is 48%. So far, so good.

This notice is to inform you that the regular network limits will go back in place at Midnight, Pacific (GMT 08:00, Tuesday) and we will go back to our billing as usual. The temporary removal of network capacity locks is a courtesy we offer our e-commerce clients so they can scale on what is the highest traffic day of the year.

Our maintenance window will be extended from 2 AM – 7 AM EST to 2 AM – 9 AM EST and will also include an afternoon maintenance interval from 4 PM – Midnight, EST on Saturday, November 24, 2007. Everything with the exception of our storage network will be fully operational, with only slight service / performance degredation that might not even be noticeable given the low network volume during Thanksgiving holiday in USA.

Upgrades are routine in nature – swapping hard drives, new memory, new switches, moving servers to optimize power utilization and cooling and other general network maintenance and management operations.