Weekly maintenance was completed on Saturday without major news to report.

Issues we are currently working on:

ExchangeDefender XD Engine Updates: We are currently monitoring the new antispam engine that went online last Wednesday. So far the performance has been remarkable and false positive ratio went down as well. You may have read about a major SPAM organization bust that went down last week, knocking McColo portion of Hurricane Electric offline, and you also may have noticed that it only affected the SureSPAM counts. This is because we have had their network mapped out for a while, the parts we are really concerned and continue to work on are the SPAM counts. You should see a huge decrease in koi8/cyrillic based SPAM messages as well as the UPS/mail trojans thanks to the new AV definition update.

Web Hosting Infrastructure: Our web hosting infrastructure will undergo a major update to address stability issues encountered over the past two weeks. Between the firewall upgrade and the major storage firmware upgrade we’ve encountered an unusual conflict that at times spikes the load on the system and sites stop responding momentarily. This is unlikely something that was noticed by our user base but it should not be happening.

Outbound traffic shaping: ExchangeDefender outbound network is being upgraded to 4.0 this week so we can bring you more accessible outbound mail tracking, smart routing and policy controls including throttling. This is an internal matter that should not affect the production network at all.

Going Concerns:

Thanksgiving holiday in United States is when we make significant changes to the network, replace a lot of aging hardware and replace with new hardware. Expect the Thursday, Saturday and Sunday of (November 27, 29 and 30) to have additional latency as we make network modifications and swaps. Details of all changes will be announced on the corporate blog since it affects all users and all services. No downtime is expected, “Black Friday” will not be affected and will again have all caps removed on traffic and quotas as our show of support for the e-commerce organizations that reside on our network.

Thank you for reading and have a wonderful day.

Management Request: If you can think of anything that could improve this site please contact us.

Extended maintenance windows over the weekend have been completed successfully and have addressed numerous growing pains and performance issues. Biggest changes were applied to our Exchange 2007 network and all the systems are now performing remarkably well. We would like to thank Microsoft CSS Exchange team for all the assistance they have rendered over the past two weeks in isolating creeping and unusual issues with the product that we have never encountered before.

Issues we are currently working on:

ExchangeDefender Reports Replication – At times the replication between our master maillog servers and slave reporting servers tends to lag. Sometimes the replication lags beyond our tolerance and appears to nearly stall. We have not yet isolated the issue that is casing this problem but are monitoring it very closely and adjusting replication points.

Offsite Backup Scaling – We are preparing our offsite backup network for another product addition to the matrix. You may experience slight availability issues as the management consoles reload while changes are taking place. This process will not interfere with the execution of offsite backups because the backup agent software automatically reloads.

ExchangeDefender SPAM Load – New ExchangeDefender antispam engine went online last week and the detection has been improved dramatically. We still haven’t kicked it up all the way but expect to by Wednesday. We have had several complaints about the level of SPAM and in our investigation have found that almost all of them are related to end user stupidity – whitelisting null senders <> and whitelisting their own domain or email address. When you whitelist any address you suppress ALL SPAM detection mechanisms, so when you whitelist your own address you open yourself up to a ton of SPAM. We have a process that we are preparing that will automatically remove null sender and mirror trusts because those should never be programmed in.

Lastly, an apology to our Canadian clients. Last week we let the cat out of the bag about our Canadian offsite backup product. This site is not intended to serve as an announcement board for new products, we just want to keep you informed about what our group is working on. From time to time what we are working on may not match the business side of Own Web Now so if you have a product or business related question we are not the most authorative source to provide an answer.

Currently there are no major problems on the deck and we have few minor going concerns related to the infrastructure. The network maintenance interval this weekend was largely uneventful.

Network Events:

Tuesday, November 4th: New self-service control panels for Microsoft Exchange 2007, Offsite Backups, Microsoft SharePoint. Invoices and service management will be launched at the same time.

Wednesday, November 5th: Introduction of the new ExchangeDefender Exchange 2007 hosting server.

Thursday, November 6th: Introduction of OWN Offsite Backup grid in Canada.

All prior issues and problems have been resolved.

Problems:

#1 DIY RBL: We are implementing new systems to help deal with the less and less responsive RBL department activity at major service providers. As a result of the SPAM problem many larger carriers have implemented their own SPAM RBL systems which have been proven less than reliable in managing SPAM. Due to the volume of messages sent by ExchangeDefender we tend to fall towards the top of the statistical profile for some of these providers and we notice issues even when there are none. At this point we are investigating complaints about Comcast. Hotmail and AT&T issuing random errors, though we are unable to successfully replicate the issue and their offices are either nonexistant (unattended web form) or don’t work during the weekends. We are working on this issue and hope the solution we are put in place is working.

#2 Too Much SPAM: ExchangeDefender SPAM levels are still higher than usual. Over the past two weeks we have shifted to our own honeypot system in addition to the external subscriptions we receive from others. Our Outlook 2007 addin allows you to submit SPAM, which virtually guarantees that you will never see that SPAM again. It took a lot of experimentation and management to get this into ExchangeDefender and we are still not at 100% on its deployment so you might see elevated SPAM levels though they should be dropping off significantly already. If you are tired of SPAM help us kill it, www.exchangedefender.com and click on New! Download ExchangeDefender Outlook 2007 Addin.

#3 Offsite Backup reports from Europe: there are still some inconsistencies successful email reports in our European offsite backup network as we roll out the new control panels and a new OSB product. Failed, missed and setting change reports from our Europe grid are coming through fine, but successful backup reports are still not being sent to 100% of the customer base. Solution to this issue is expected today and we are working on a workaround.

Solutions:

Issue with successful job Offsite Backup reports in USA has been resolved.

Issues related to Exchange 2007 back pressure has been resolved.

Issues with BT have been resolved.

Workarounds:

We will adding an adaptive feature for outbound mail routing to address future #1 issues. This will be announced somewhere towards the end of the week and will require an adjustment to your SPF records if you use that technology. Because large ISPs do not use the same RBLs that rest of us use we have no ability to monitor them and proactively address issues. Going forward we will do manual monitoring and route mail through the IPs that do not appear to be affected by a single provider.

Going Concerns:

Tonight we will be rolling out master control panels for OWN services. This will give you full control over accounts and settings for our Microsoft hosting and Offsite backup services with centralized management. As with any software rollout there could be issues, we will be staffing extra support to address them all.

New Exchange grids coming online this week. We will be adding another Exchange 2007 cluster to the service which will require some documentation adjustments and modifications.

Problems:

We are still working on British Telecom which is rejecting messages with Relaying Denied. In the meantime please create a separate SMTP connector and attempt to deliver messages to them directly. We are still working with them on daily basis to resolve the issue but their response so far has not led to a solution.

ExchangeDefender Outlook 2007 Addin is being improved towards 1.1 and fixes are being rolled up as well. Currently most issues reported have to do with 64bit Vista. Other issues are noted in the documentation (for example, do not install out of a zip file, do not install from a UNC/network share).

Shockey Monkey bugfix update is scheduled for this Saturday, 10/18/2008, with changes to the reports and email notification upgrades.

Solutions:

Offsite Backups reports are now functioning 100% across the client base. The issue of some reports (backup pass/fail) have been resolved by AhSay with upgrade to 5.5.1. You do not need to upgrade your agents to take advantage of this fix, the issue was on the servers not agents.

ExchangeDefender email reports issue has been fixed. There was a brief sync issue between the report servers and admin servers which resulted in delayed updates to the configuration changes. This has been addressed but we are also working on a completely different algorithm that can compensate for SQL replication issues.

Exchange 2007 backpressure problems have been solved by removing the feature on all servers.

Going Concern:

Tuesday/Wednesday nightmares continue this week with our Microsoft network. We are expecting 11 updates this week that we will be rolling out within 24 hours of successful testing.  Although none of the updates affect the server-side of the network (all are related to Internet Explorer and MBSA) we have a policy of keeping our servers up-to-date and will be applying the patches silently.

We have still not completed our evaluation of Exchange 2007 SP1 Update Rollup #4. The problems that we have reported still exist in the new code and at this point we do not see a reason to upgrade. Here are the outstanding issues we are tracking with Exchange 2007:

1. iPhone sync crashes store
2. Mailbox database dismounts without event log notification, service remains in running state.
3. Store service hangs.
4. Backpressure reports invalid data and does not respect treshhold definitions.

ExchangeDefender - Solid. No issues, no latencies, no delays and overall flawless performance. Over the past week we have not had any delay reports that turned out to be on this end, we have had no DDoS issues to report, we have our lowest latency ever (between 5-10 seconds, network-to-network) and fewest false positives in a while.

Hosting - Solid. No issues, working on Exchange 2007 SP1 upgrades and 2003-2007 migrations as well as WSS2 to WSS3 migrations. We are working on integrating our new global points of presence in Australia and Europe and they are coming together well.

Offsite backups - Solid. No major issues, one patch and hardware enhancement scheduled for this weekend to help with large file backups.

VoIP - We are adding another IAX2 provider to our network, should become available shortly.

Training - Mostly scenario and interface tests for ExchangeDefender 4.0. Bringing on new materials, including Shockey Monkey 2.0. Web site undergoing a major update.

That is all. It’s so boring it already feels like the summer.

For what it’s worth, we realize that this is getting ridiculous. Unfortunately, this one is out of our control.

We have been planning this page for you, our customers and partners, for quite some time but decided to rush it due to the amount of feedback we received from you. So, what can you expect from this site? Constant, constant, constant network updates on availability, maintenance, emergencies and up-to-the-minute advisories on everything we do.

I want to ask for a favor. Subscribe to the global feed if you can but since we don’t want to overwhelm you with the amount of stuff going on at Own Web Now, I would like you to separately subscribe only to the services that you actually purchase from us. I have tried to boldface the important categories to the left, I hope they prove worthwhile to you.

This site also means the pending death of Own Web Now Corp email alerts. With the exception of the most important service notices, everything will be posted here instead of via email. If you are an “email person” and don’t like RSS feeds please check out Feedblitz, which converts RSS feeds to emails! While we cannot guarantee the availability or timeliness of Feedblitz I still wanted you to be aware of it in case it makes it more convenient for you to follow this site.

So, that is all. Thank you for looking around. Thank you for relying on our services. As I said, we did have to get this out ahead of schedule so you will see us bring more and more information to this blog as well as full integration into OWN Support portals, ExchangeDefender Administrator & SP Control Panels, etc. While this will be the central storage point of all alerts, they will be syndicated to the above mentioned sites.

Have a great day!

OWN NOC Team