Services Partners Contact About Help Support Blog Login

Network Operations

LiveArchive Service Restored
Posted: 12:11 pm
September 15th, 2008
 
ExchangeDefender

We have restored the service to the small portion of users that were affected by its outage this morning. The issue had to do with a hotfix provided by Microsoft. Hope everyone in the midwest and Ohio weathers through the storms that have knocked much of the power out in that region.



Sounds Like Someone Has a Case of the Mondays
Posted: 1:04 pm
September 8th, 2008
 
Data Center Ops, ExchangeDefender, Offsite Backups

We must have angered the Internet gods because this Monday has been nothing short of tremendously disappointing. Pictured below is my staff working on the issues:

office-space

On to the specifics:

ExchangeDefender reports did not run last night and will likely remain offline until close of business today. We have had two switch crashes on our load balancers in front of our shared mail1 and www1 hosting services. Our offsite backup upgrade does not seem to be validating the certificate requests so https:// requests are failing (http:// still works fine, and data is encrypted on the client side so the transport mechanism isn’t as relevant - but if you’ve set https:// your backups are failing so we are treating this as a very serious issue)

Somehow, the roof is still above us and we have power. For now.

All the outstanding issues are being filtered through by my teams and will have service restored to 100% across the entire product portfolio - by the end of business today.

Update: As of 5 PM EST the ExchangeDefender reporting is back online, all the network issues have been resolved. The Offsite Backup service is still available via http:// but we are still working with AhSay to get the certificate issue resolved. Will update further on this as soon as I have more information.

Update: As of 11 PM EST all offsite backup grids now respond with the valid SSL certificates on the SSL port.

Looks like the ugly Monday is finally behind us.

Sincerely,

Vlad Mazek, CEO



Verizon Blacklist
Posted: 11:42 am
September 5th, 2008
 
ExchangeDefender

We have received several reports this morning about our IP address blocks being on Verizon’s RBL. The following errors were given to some of our customers on ExchangeDefender:

outbound1.exchangedefender.com #5.5.0 SMTP; 571 Email from 65.99.255.236 is currently blocked by Verizon Online’s anti-spam system. The email sender or Email Service Provider may visit http://www.verizon.net/whitelist and request removal of the block.>

In our calls and discussions with Verizon we have received a confirmation that we are not and have not been on their RBL. At this point the mail is routing correctly so we are just chalking this up to there being a temporary glitch with Verizon’s RBL systems.



Reports Email Issues
Posted: 11:17 pm
September 3rd, 2008
 
ExchangeDefender

Over the past two days that the reports service has been restored we’ve discovered a few bugs in the system that prevented proper delivery and branding of the SPAM reports. Even though they were generated properly, the reports got routed through the ExchangeDefender inbound network instead of direct to the servers. This unfortunately may have gotten trapped in the junk mail again.

This issue was corrected at 11 PM EST (4 AM GMT).



Report Services Offline
Posted: 12:14 am
September 1st, 2008
 
ExchangeDefender

We are talking advantage of an extended holiday weekend in United States to perform network upgrades and maintenance as well as a software rollout on our email reporting grid for ExchangeDefender. We have rolled in ExchangeDefender 4.0 upgrades to this system and are taking an extra day to put it through it’s paces and make sure it’s 100% solid.

For our customers abroad that will be affected by the email reports please keep in mind that this legacy system is just one of the ways to access junk. The recommended and preferred way of accessing SPAM quarantines for ExchangeDefender is the web portal at https://admin.exchangedefender.com and we also offer the SPAM Monitor desktop software with hourly alerts.  We anticipate regular daily reports to resume on Tuesday.



Blacklist Monday - Verizon, FrontBridge
Posted: 12:51 pm
August 18th, 2008
 
Uncategorized

Good morning - we are currently investigating two blacklists from large ISPs targeting one IP address on the ExchangeDefender network. Two services are identified as Verizon and FrontBridge and we have opened requests to be removed from both along with any data that might help us find out why the lists were put in place to begin with given our 0 tolerance for SPAM. The rejections are marked by:

< mail157-dub-R.bigfish.com #5.0.0 X-Postfix; host    winse-6216mail6.customer.frontbridge.com[131.107.115.215] said: 550 5.7.1    External client with IP address 65.99.255.232 does not have permissions to  submit to this server. Visit http://support.microsoft.com/kb/928123 for more information. (in reply to end of DATA command)>

 

<outbound2.exchangedefender.com #5.5.0 SMTP; 571 Email from 65.99.255.232 is currently blocked by Verizon Online’s anti-spam system. The email sender or Email Service Provider may visit http://www.verizon.net/whitelist and request removal of the block.>

In the meantime, you can change your outbound smarthost to outbound1.exchangedefender.com if you experience the problems above.

On the funny side: We find it hilarious that Microsoft is linking to their own KB articles in the rejection note for a problem that is caused on their own servers - how about something more helpful like postmaster or delisting contact address or URL! Even more surprising is that Microsoft FrontBridge is running on an open source Postfix mail platform, not a Microsoft one.

We will update you on the delisting process as we get more information. The problem should not impact many senders as 65.99.255.232 is just one of the nodes in our outbound network.



New ExchangeDefender grid in action
Posted: 12:47 pm
August 13th, 2008
 
ExchangeDefender, Network Operations

Earlier today we completed the rollout of 450 new servers to the ExchangeDefender family all over our American network. The introduction and initial sync of the new nodes did allow some junk through as well as introduce a slight today (maximum reported 1 hour from one system that nearly immediately went into maintenance mode) but as of roughly 11:30 AM EST all is good.

Additional 600 nodes are planned in our global expansion leading up to ExchangeDefender 4.0 launch. We are also looking at additional data centers on both coasts at the moment scheduled to go live this fall.

Update: 2:24 PM EST: We are happy to report that all the nodes have now converged in the scanning network and the SPAM filtering is back at its usual levels (and to be tightened up even further later tonight). You may have seen an increase in SPAM over the past few hours while the nodes were joining the network and accepting new programming but you should be seeing far less SPAM going forward.



Rise in SPAM reported from Europe
Posted: 9:24 am
August 11th, 2008
 
ExchangeDefender

We have several reports from our UK and Ireland customers of the rise in the amount of junk mail passed through ExchangeDefender this morning. Aside from a strain of CNN-forged SPAM we are not seeing any issues in ExchangeDefender nor do our stats show anything out of the ordinary at the moment. We are investigating the situation.

The SPAM regarding CNN is already in the filters and should be stopped going through further. For anything else that may slip through please forward the message with SMTP headers to spam@ownwebnow.com and we will gladly investigate it.

Update: We had a rule update that unfortunately offsite all the other CNN rules and let that junk through. The team is now filtering it through both the pattern search and hyperlink drop on the domains used to get traffic. We are seeing a few other SPAM strains getting more popular today as well (Wall Street Subscription scam, fake MSN alert to download Internet Explorer 7). All of these are now effectively being filtered by ExchangeDefender which undergoes thousands of updates a day but due to the CNN rules that have been changing a lot over the past few days, and in light of the six complaints we got this morning, we felt it was important to update in more detail than usual.

Update 2: We are seeing things under more and more control as we continue to filter out the strains of the three major junk items. As a matter of policy we do not publish our filtering technology or keywords or scores but we are currently tracking the variants of CNN, WSJ, Internet Explorer 7 and a few smaller ones.



Los Angeles Earthquake
Posted: 3:00 pm
July 29th, 2008
 
Data Center Ops

As you may be aware, we have two data centers in Los Angeles on Wilshire Blvd. Earlier today, this area suffered a 5.8 magnitude earthquake. No systems were affected, no impact on any power feeds or network connections. Earthquakes tend to be followed by smaller “aftershocks” and we will be updating this post with details of any relevant information that may become available.



ExchangeDefender and Virus Warnings
Posted: 8:37 am
July 28th, 2008
 
ExchangeDefender

Over the weekend we tested and perfected a new method for managing archive embedded dangerous content. During the deployment of the new software some archives were improperly classified as dangerous and archives (.zip, .arj) removed. That issue has been solved as of Sunday evening.

As a point of reference, ExchangeDefender does not allow executable attachments (.exe, .bat, .com or .pif) in either standalone or archived mode. That means even if you zip the file up it will be picked up by a scanner. If you zip a zip file, the system will reject to process it. This has been our long standing tradition of not allowing dangerous content through the network because virus scanners sometimes do not react as quickly to the rise in malware and our responsibility is to protect our customers. If you need a dangerous attachment really bad, for the safety of the less IT savvy people in your organization, please try to find alternate means such as a web sharing tool or a freemail account.

We have also addressed this need in ExchangeDefender 4.x which is scheduled for August 19th.



Exchange 2007 Global Network Update
Posted: 4:48 am
July 9th, 2008
 
Exchange Hosting

Following is our reboot cycle for our global Exchange 2007 network. Microsoft recently published the Exchange Server 2007 SP1 Update Rollup 3, SQL Server Service Pack 2 and several security patches that have been tested by OWN and approved for rollout. Scheduled reboots are as follows:

USA - 5 AM EST (11:00 GMT) Today

Europe - 5 AM GMT Tomorrow

Australia - Midnight EST-Australia Today (10 AM EST United States)

We do not anticipate any outage or issues with the patches, they have passed lab tests without problems.



ExchangeDefender processing delays
Posted: 4:29 pm
June 23rd, 2008
 
ExchangeDefender

We are currently addressing a processing delay in ExchangeDefender antivirus scanning engine. One of our virus engine vendors had distributed a faulty update which has caused a backlog of messages that have been quarantined for further inspection.

The ExchangeDefender system is designed to apply stronger scrutiny and more intensive checks against any attachments or messages that have produced any sort of an error in any of our antivirus scans. The reason we run multiple scanner engines is because not all engines are as thorough or as rapidly updated as the threats emerge and change in the wild. Once an issue is encountered we scan with more options to find out if the message is indeed dangerous or if there is something wrong with a portion of it (attachment, envelope).

In this case the corrupt message was passed on to ExchangeDefender which quarantined messages for further scanning which is far more expensive and processor intensive. We have responded immediately and removed the engine, however, even slight issues can cause huge problems when you process as much mail as we do and it has introduced a slight delay in the processing of messages. The issue started at roughly 3:10 and was resolved by 3:40. At the time of this message we see around 60% of our nodes processing messages within our ordinary SLA (seconds) and we expect the rest of the network to catch up shortly.

If you experience any delays, even extensive in nature, it is due to the above problem which will within 30 minutes be completely under control.



Maintenance on the Virtual and Web hosting Windows Network
Posted: 11:10 pm
June 22nd, 2008
 
Virtual Hosting, Virtual Servers

We are conducting some routine maintenance on our Windows portion of the Virtual Hosting and Web Hosting network. We are applying patches, installing new hardware, general system maintenance task. All systems should be affected by a brief outage, and will be back within an hour at most.

Update: Maintenance cycle completed.



Exchange 2007 OWA
Posted: 2:30 pm
June 10th, 2008
 
Exchange Hosting

We are currently addressing an issue with Exchange 2007 OWA. You will see the following text when attempting to login:

Outlook Web Access did not initialize. An event has been logged so that the system administrator can resolve the issue. Please contact technical support for your organization.

Request
Url: https://scrooge.exchangedefender.com:443/owa/auth/error.aspx

We will have this addressed momentarily and update this site.

Update: 2:45: Problem has been solved.



Offsite Backup Maintenance
Posted: 9:36 am
May 14th, 2008
 
Offsite Backups

We are conducting maintenance on our Offsite Backups architecture. 08:00–14:00 EST is our slowest time of the day and we’ll have the systems back online in time for the nightly backups.



Issues on outbound.exchangedefender.com
Posted: 9:03 am
May 12th, 2008
 
ExchangeDefender

Earlier today we had to flush the queues on ExchangeDefender outbound server due to the large number of corrupt queue files sent by one of our customers malfunctioning servers. If your messages were not delivered during the window between 5am - 7 am central (GMT -6) please resend them.

The problem has been solved temporarily, but we will be holding an urgent maintenance window this Wednesday, 5/14, to address the core of the problem.

P.S. Significant number of servers were backlogged during this process. That mail has been processed without issue.



Tracking issue on shared mail server mail1
Posted: 1:13 pm
May 8th, 2008
 
Virtual Hosting

We are currently tracking accessibility problems on mail1.ownwebnow.com. Please stand by while we research the issue, the server appears up but several customers are reporting access issues, we are trying to resolve them right now.

We will update this site as soon as we have more information. The cluster is currently undergoing a reboot.

Update: All issues have been resolved, a scheduler service hung on the load balancer.



Network event in Dallas
Posted: 7:54 am
April 29th, 2008
 
Network Operations

We are currently investigating a network event in our Dallas region network centers, since approximately 5 AM EST we have been receiving complaints about network connectivity and availability. There are currently no outages and there have been no outages but certain customers are unable to reach the services on our network.

If you are experiencing an issue, please open a support request and include a traceroute to the service you are trying to reach (ex: 65.99.255.50).

We will update this ticket as soon as we have further information. 

Update (9:05 AM EST, 14:05 GMT): We believe the network issue some of our customers have experienced has been resolved. Particularly affected were some of our UK customers (not BT) and local customers with Level 3 connectivity. Although we see the traffic back up at usual levels, it may take about an hour or so until all the mail catches up and gets delivered.



Investigating problems with roadrunner network
Posted: 1:40 pm
April 23rd, 2008
 
ExchangeDefender, Virtual Hosting

We are currently working with RoadRunner (formerly Time Warner, AOL) service provider in United States, they are experiencing issues with their SMTP servers and randomly rejecting SMTP traffic. Currently mail is flowing through but some is bouncing back from them due to a reason they are still trying to narrow down. We will update when we have further information or a resolution.

This issue affects our entire global network, and some external sites we have tested.

Update: 6:34 PM EST: Even though we have not been officially updated, the problems with RoadRunner appear to have been resolved. 



Exchange 2007 SP1 upgrade underway
Posted: 8:01 am
April 6th, 2008
 
Exchange Hosting

We will be starting the SP1 upgrade on our systems in roughly two hours (10 AM EST, -5 GMT) and expect to have all operations completed by noon. To minimize the surprises and potential conflicts, the entire cluster and all its members will be patched at once. That unfortunately does mean a bit of total outage, but it does minimize the chance of anything breaking in the process. Hosted Exchange has been one of our most solid products and we look forward to keeping it that way.

Update: Exchange 2007 Service Pack 1 is now up and running, systems have been imaged and we are all done.



Own Web Now Blog

Own Web Now Corp Network Operations Center is designed specifically to alert and inform you of routine and emergency maintenance tasks being conducted on our network. Every large product group is represented in the feeds below and we urge you to follow the global alert feed or at least the individual service feeds to which you subscribe.


News & Events

The best way to stay in touch with us is through our blog, but from time to time we do special things that we feel you might find interesting. Check them out!

July 17, 2008
OWN Partner Call 5
Dana Epp talks about a new security solution offering for SMB.

Alternative content



July 11, 2008
OWN Partner Call 4
Matt Makowicz talks about maturity of the partner business at WPC.

Alternative content