We are currently conducting an out-of-band security patch on mail1.ownwebnow.com network. We regret to make this modification during business hours but the scope of security issue was wide enough to warrant immediate action. Service has been restored to 100% at 4:48 PM EST.
Good morning and happy Monday! It’s that time of the week again, where we open up our NOC task list and sprinkle some pixy dist on the wishes that the entire network remains at 100% uptime. Weekly maintenance was completed on Saturday without major news to report. Issues we are currently working on: ExchangeDefender XD Engine Updates: We are currently monitoring the new antispam engine that went online last Wednesday. So far the performance has been remarkable and false positive ratio went down as well. You may have read about a major SPAM organization bust that went down last week, knocking McColo portion of Hurricane Electric offline, and you also may have noticed that it only affected the SureSPAM counts. This is because we have had their network mapped out for a while, the parts we are really concerned and continue to work on are the SPAM counts. You should see a huge decrease in koi8/cyrillic based SPAM messages as well as the UPS/mail trojans thanks to the new AV definition update. Web Hosting Infrastructure: Our web hosting infrastructure will undergo a major update to address stability issues encountered over the past two weeks. Between the firewall upgrade and the major storage firmware upgrade we’ve encountered an unusual conflict that at times spikes the load on the system and sites stop responding momentarily. This is unlikely something that was noticed by our user base but it should not be happening. Outbound traffic shaping: ExchangeDefender outbound network is being upgraded to 4.0 this week so we can bring you more accessible outbound mail tracking, smart routing and policy controls including throttling. This is an internal matter that should not affect the production network at all. Going Concerns: Thanksgiving holiday in United States is when we make significant changes to the network, replace a lot of aging hardware and replace with new hardware. Expect the Thursday, Saturday and Sunday of (November 27, 29 and 30) to have additional latency as we make network modifications and swaps. Details of all changes will be announced on the corporate blog since it affects all users and all services. No downtime is expected, “Black Friday” will not be affected and will again have all caps removed on traffic and quotas as our show of support for the e-commerce organizations that reside on our network. Thank you for reading and have a wonderful day. Management Request: If you can think of anything that could improve this site please contact us.
AT&T is having RBL issues again, we are working with them to resolve the problem. You may receive this problem when emailing the AT&T network for the time being: <<< 521-65.99.255.236 blocked by sbc:blacklist.mailrelay.att.net.<<< 521 DNSRBL: Blocked for abuse. See http://att.net/blocks554 5.0.0 Service unavailable We have put in place a workaround and are working with AT&T to resolve the issue. You should not continue to see this problem. However, the issue is still open.
We are investigating issues with hosted exchange 2007 servers in Dallas at this moment. The service is currently being restored and should be back to normal momentarily.
We had an authentication issue reported on daisy.theofficeserver.com Exchange 2003 hosting platform. The issue was related to an older SSL certificate binding to the wrong IP address and stopping authentication and OWA requests that check for the certificate validity. Issue was promptly corrected and old certificate permanently removed. Thank you and have a great day.
Good morning and happy Monday! This posting is part of our effort to keep you more in tune with the network operations group at Own Web Now because you rely on our services and should be a part of the conversation. Extended maintenance windows over the weekend have been completed successfully and have addressed numerous growing pains and performance issues. Biggest changes were applied to our Exchange 2007 network and all the systems are now performing remarkably well. We would like to thank Microsoft CSS Exchange team for all the assistance they have rendered over the past two weeks in isolating creeping and unusual issues with the product that we have never encountered before. Issues we are currently working on: ExchangeDefender Reports Replication – At times the replication between our master maillog servers and slave reporting servers tends to lag. Sometimes the replication lags beyond our tolerance and appears to nearly stall. We have not yet isolated the issue that is casing this problem but are monitoring it very closely and adjusting replication points. Offsite Backup Scaling – We are preparing our offsite backup network for another product addition to the matrix. You may experience slight availability issues as the management consoles reload while changes are taking place. This process will not interfere with the execution of offsite backups because the backup agent software automatically reloads. ExchangeDefender SPAM Load – New ExchangeDefender antispam engine went online last week and the detection has been improved dramatically. We still haven’t kicked it up all the way but expect to by Wednesday. We have had several complaints about the level of SPAM and in our investigation have found that almost all of them are related to end user stupidity – whitelisting null senders <> and whitelisting their own domain or email address. When you whitelist any address you suppress ALL SPAM detection mechanisms, so when you whitelist your own address you open yourself up to a ton of SPAM. We have a process that we are preparing that will automatically remove null sender and mirror trusts because those should never be programmed in. Lastly, an apology to our Canadian clients. Last week we let the cat out of the bag about our Canadian offsite backup product. This site is not intended to serve as an announcement board for new products, we just want to keep you informed about what our group is working on. From time to time what we are working on may not match the business side of Own Web Now so if you have a product or business related question we are not the most authorative source to provide an answer.
We will be conducting a major maintenance window this Sunday, November 8th, 2008. We will be deploying series of hotfixes provided by Microsoft for a slew of bugs Own Web Now Corp has reported over the past six months. We have also received a lot of guidance in the way of optimizing our setup and will with Microsoft’s help proceed to make major adjustments to the platform. Unfortunately, this means that some users may experience issues during Sunday early AM hours. Although our systems are clustered some changes require database moves and service restarts which will have to be done in sync and will unfortunately lead to service interruptions. Our goal as always is to keep these service interruptions to the minimum and limit them to maintenance hours, however, since these issues will be sporradic throughout the night we wanted to note them here. After the initial test on our own Exchange 2007 network we will be applying the same fixes and optimizations to our dedicated server clients running Exchange 2007. ExchangeDefender will not be impacted, however, your mail may experience slight delay if you are on Exchange 2007 mailbox store which is being cycled and ExchangeDefender is not able to immediately deliver the message. In this case we recommend all our mission critical 24/7 operations to fall over to LiveArchive which will be available. Over the past 12 months we have had a 99.999% uptime on our Exchange 2007 network and 100% uptime on our ExchangeDefender network. Those numbers are impressive but only possible thanks to preventive maintenance and optimizations as noted above. We apologize in advance for any inconvenience you experience during the maintenance cycle.
We have been made aware of an issue with email reports showing all 0’s for SPAM stats. The issue has been resolved as of 11:30 AM EST. Please note that we do not recommend using email reports and encourage everyone to migrate to the new methods of accessing SPAM: realtime web portal, desktop agent or Outlook 2007 agent.
Another in the series of Monday posts covering the network events at Own Web Now. Currently there are no major problems on the deck and we have few minor going concerns related to the infrastructure. The network maintenance interval this weekend was largely uneventful. Network Events: Tuesday, November 4th: New self-service control panels for Microsoft Exchange 2007, Offsite Backups, Microsoft SharePoint. Invoices and service management will be launched at the same time. Wednesday, November 5th: Introduction of the new ExchangeDefender Exchange 2007 hosting server. Thursday, November 6th: Introduction of OWN Offsite Backup grid in Canada. All prior issues and problems have been resolved.
Due to the latest vulnerability in Microsoft Windows software and the vulnerability already remotely exploitable we are forced to do server reboots of our entire Windows Server network outside of our regularly scheduled maintenance interval. We apologize for any inconvenience this may cause you.
At approximately 4AM EST we have noticed a failure in updates from one of our AV vendors. That failure produced higher than expected virus matches which ended up queuing a larger than normal amount of messages. We have resolved the issue with the update and are currently re-processing all the mail that was quarantined over the past few hours. Please stand by, we will deliver all mail. Update: 7:29 AM EST: Nearly all the mail that was affected by the faulty AV update has been processed and has been dispatched to delivery queues. As of the previous update, all new mail has been delivered in realtime. It is important to note that we are only processing the backlog for the messages that did get trapped by the faulty AV update. Update: 9:15 AM EST: 99% of the messages have been flushed out. By the time you read this posting all the mail would have been delivered. No mail has been dropped during the period, if you experience further issues with delays please follow our deployment guide and support documentation, we find most delays are related to the on-premise issues relating improper firewall configuration, connection rate limiting (by far) and other SPAM/malware scanning that does not properly whitelist ExchangeDefender systems.
We are continuing the “Case of Mondays” posts covering OWN infrastructure events we are working on so you can be more connected with Own Web Now. On behalf of the entire team thank you for the nice emails you’ve sent to us regarding this. As a result of such tremendous feedback will soon be opening a forum to continue the conversation throughout the week. Problems: #1 DIY RBL: We are implementing new systems to help deal with the less and less responsive RBL department activity at major service providers. As a result of the SPAM problem many larger carriers have implemented their own SPAM RBL systems which have been proven less than reliable in managing SPAM. Due to the volume of messages sent by ExchangeDefender we tend to fall towards the top of the statistical profile for some of these providers and we notice issues even when there are none. At this point we are investigating complaints about Comcast. Hotmail and AT&T issuing random errors, though we are unable to successfully replicate the issue and their offices are either nonexistant (unattended web form) or don’t work during the weekends. We are working on this issue and hope the solution we are put in place is working. #2 Too Much SPAM: ExchangeDefender SPAM levels are still higher than usual. Over the past two weeks we have shifted to our own honeypot system in addition to the external subscriptions we receive from others. Our Outlook 2007 addin allows you to submit SPAM, which virtually guarantees that you will never see that SPAM again. It took a lot of experimentation and management to get this into ExchangeDefender and we are still not at 100% on its deployment so you might see elevated SPAM levels though they should be dropping off significantly already. If you are tired of SPAM help us kill it, www.exchangedefender.com and click on New! Download ExchangeDefender Outlook 2007 Addin. #3 Offsite Backup reports from Europe: there are still some inconsistencies successful email reports in our European offsite backup network as we roll out the new control panels and a new OSB product. Failed, missed and setting change reports from our Europe grid are coming through fine, but successful backup reports are still not being sent to 100% of the customer base. Solution to this issue is expected today and we are working on a workaround. Solutions: Issue with successful job Offsite Backup reports in USA has been resolved. Issues related to Exchange 2007 back pressure has been resolved. Issues with BT have been resolved. Workarounds: We will adding an adaptive feature for outbound mail routing to address future #1 issues. This will be announced somewhere towards the end of the week and will require an adjustment to your SPF records if you use that technology. Because large ISPs do not use the same RBLs that rest of us use we have no ability to monitor them and proactively address issues. Going forward we will do manual monitoring and route mail through the IPs that do not appear to be affected by a single provider. Going Concerns: Tonight we will be rolling out master control panels for OWN services. This will give you full control over accounts and settings for our Microsoft hosting and Offsite backup services with centralized management. As with any software rollout there could be issues, we will be staffing extra support to address them all. New Exchange grids coming online this week. We will be adding another Exchange 2007 cluster to the service which will require some documentation adjustments and modifications.
A number of users have reported inability to receive email. Sending email works fine but receiving does not. In multiple tests against multiple sites, both protected by ExchangeDefender and not, we have seen nearly 60% connection drops. At this point we believe this is related to the latest Microsoft patches - and a simple reboot appears to clear out the issue of no port 25 connectivity. Don’t worry, ExchangeDefender is holding your email. It does not appear to be fixed by removing the patches and reapplying them. As our friend Susan Bradley just told me via IM: “Full moon passing” Vlad Mazek, MCSE CEO, Own Web Now Corp
As you may have noticed over the past few weeks, the SPAM levels have increased slightly. Unfortunately, even a slight increase in the SPAM levels as a percentage can result in getting a piece or two an hour as opposed to a piece or two a day. Yesterday we finally isolated the issue that was causing this thanks to a few of our partners and the new ExchangeDefender Outlook 2007 addin. We are still working on automating the distribution and monitoring of the new processes that will keep this from coming up again. Further Details ExchangeDefender has multiple grids around the world. All grids use a central RBL distribution database that is centrally managed and monitored. Every grid has it’s own DNS caching servers that hold both the RBL data as well as our clients IP address information for delivery, routing and SPAM definitions. Since the latest update to our core distribution the DNS server performance has been flaky and would simply stop returning results. Because our RBL code is set to look for matches in the RBL zone the servers lack of response, or lack of correct response, means that the messages that were certainly SPAM were allowed to go through the less-restrictive SPAM scanning and unfortunately that contributes to 1-2% difference in the SPAM load and in some cases latency for nodes that are about to go into the shutdown/maintenance mode and are flushing out their queues. Because ExchangeDefender delivery queues run off the same DNS infrastructure (technical limitation) this compounds the problem and issues as the resolutions do not come from the primary (on-node) or secondary (on-grid) but a tertiary (central OWN NOC) DNS server. What we have done so far is implementing a system that does local resolver check and restarts the DNS service if it is not returning proper data. What we are currently working on is a monitoring system to centrally report the issues with the resolver latency (one of the things we currently do not measure) as the lookups have to skip to the secondary or tertiary systems. We expect to have all the issues handled by the end of the weekend. From statistical breakdowns we know that the issue has not been widespread (only certain users would even have noticed the difference) and only about a dozen people have complained so far. Unfortunately for us, the people likely to notice are the people that get the most mail and the ones that likely love our product the most. We’ll get this one taken care of for you folks, thanks for your patience.
As a continued effort to keep you completely in the loop of all the events at Own Web Now we will start a new post series called “Case of Mondays” where we discus ongoing issues with network and any changes that are being made throughout the weekend Maintenance schedules to keep the systems running along as well as any issues that we are working on that might come up. Problems: We are still working on British Telecom which is rejecting messages with Relaying Denied. In the meantime please create a separate SMTP connector and attempt to deliver messages to them directly. We are still working with them on daily basis to resolve the issue but their response so far has not led to a solution. ExchangeDefender Outlook 2007 Addin is being improved towards 1.1 and fixes are being rolled up as well. Currently most issues reported have to do with 64bit Vista. Other issues are noted in the documentation (for example, do not install out of a zip file, do not install from a UNC/network share). Shockey Monkey bugfix update is scheduled for this Saturday, 10/18/2008, with changes to the reports and email notification upgrades. Solutions: Offsite Backups reports are now functioning 100% across the client base. The issue of some reports (backup pass/fail) have been resolved by AhSay with upgrade to 5.5.1. You do not need to upgrade your agents to take advantage of this fix, the issue was on the servers not agents. ExchangeDefender email reports issue has been fixed. There was a brief sync issue between the report servers and admin servers which resulted in delayed updates to the configuration changes. This has been addressed but we are also working on a completely different algorithm that can compensate for SQL replication issues. Exchange 2007 backpressure problems have been solved by removing the feature on all servers. Going Concern: Tuesday/Wednesday nightmares continue this week with our Microsoft network. We are expecting 11 updates this week that we will be rolling out within 24 hours of successful testing. Although none of the updates affect the server-side of the network (all are related to Internet Explorer and MBSA) we have a policy of keeping our servers up-to-date and will be applying the patches silently. We have still not completed our evaluation of Exchange 2007 SP1 Update Rollup #4. The problems that we have reported still exist in the new code and at this point we do not see a reason to upgrade. Here are the outstanding issues we are tracking with Exchange 2007:
We have received several reports of issues with BT. You may receive this error when sending messages to btinternet.com recipients.
We have notified BT by e-mail and phone regarding the issue, the problem is on their end. Since this is a configuration issue on BT network we have no ETA, no resolution time or idea of what may be going wrong. For more information about proxy errors, click here.
We are currently tracking issues that have been reported by multiple users:
We will update as we get more information.
Over the past week or so we have been tracking sporadic complaints about mail delivery delays to Exchange 2007 hosting. It appears that in rare cases the Exchange 2007 network temporarily rejected the message with a 4.3.1: Insufficient system resources message even though plenty of resources were available on the system to process the message and plenty of storage was available on the volume. We have seen this issue creep up from time to time on multiple Exchange 2007 issues and have provided information to Microsoft regarding a possible bug but have so far been unsuccessful in getting a response because the issue is so sporadic that it cannot be pinned down to a single factor and only a small number of messages encounters an issue. What is even more frustrating is that even during heavy maintenance and optimization tasks the issue does not creep up where it is expected with back pressure controls. This Sunday we ran a database optimization maintenance process and out of roughly 1,000 messages processed less than 20 were deferred with the Insufficient system resources error even though the CPU utilization was above 80%. Due to the level of complaints and our inability to isolate this issue with Microsoft we have disabled the back pressure feature of Microsoft Exchange 2007 on all our systems until further notice. Further details about Exchange 2007 and back pressure is available at Microsoft TechNet. Note: This issue is isolated to the new feature in Exchange 2007 called back pressure. Exchange 2003, ExchangeDefender and ExchangeDefender LiveArchive are not affected by this issue.
We are currently performing maintenance on the USA offsite backup infrastructure to add more capacity to the logging partitions. The service will be restored today and emergency restore access is still available. This issue does not affect Offsite Backups Europe.
We will be extending our maintenance window for the report services this weekend in order to implement the new ExchangeDefender 4.0 functionality. While the reporting should not be impacted during this time, our support teams will have limited visibility to the backend and might not be able to effectively troubleshoot the issues. We are sorry for any inconvenience this might cause your clients but we’re confident you will be pleased with the results. Maintenance: Sunday, 1 AM EST - 6 AM EST. |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
