We are currently addressing an issue where the webmail site for mail1.ownwebnow.com (Webhosting mail server). We do not anticipate the outage lasting more than 20 minutes.

Update 11:23 AM Eastern: We’ve detected a hardware issue on mail1. In the interest of uptime during peek hours for our customers, we’ve decided to hold off on fully fixing the issue until tonight. Throughout the day, customers may experience brief periods of inaccessibility as the server soft-recovers itself.

Our shared POP3 mailbox server mail1 is currently under an extreme load and slowly processing messages. We are working to alleviate the queue load and restore mail routing.

Update 1:11 PM: The load on the mail1 server has been reduced and new mail is being processed successfully. We anticipate issues for some users connecting as all users are concurrently connecting to download mail.

We are currently addressing a hardware issue on the Linux hosting cluster, affecting web, ftp and sql services. We will update the ticket when significant progress has been made, at this time we do not have an ETA as the file systems are being checked for errors.

Update: 6 PM EST: At this point over 90% of the service has been restored. We are moving on to more complex, database driven systems at this point.

Update: Saturday 8 PM EST: At this point all sites, even complex SQL driven ones, are online. Service upgraded to new versions of SQL and PHP and a more robust flexible storage array.

Earlier today we encountered several tempfail mail delay scenarios due to a piece of software relying on DSBL realtime blacklist that was no longer actively maintained. We have taken action to remove DSBL from our web hosting mail servers which has resolved any delays and is currently processing the mail backlog.

Note: The issue only applied to mail1.ownwebnow.com. It did not affect any other services, such as Exchange hosting or ExchangeDefender. The issue was resolved and there should be no delays on the network at this time.

We are currently investigating an issue on one of our pop3 hosting servers. It appears to be a complex issue affecting a number of accounts. We believe we are near bringing this component back to full functionality.

Symptom: Cluster is up and responsive but certain users are not receiving mail. (related to a corrupted journal on one of the SANs)

11 AM EST: Service restored. We are still researching the issue and will provide an update after the weekly maintenance this Saturday. This appears to have been caused by a DDoS that did not get properly picked up.

We are currently conducting an out-of-band security patch on mail1.ownwebnow.com network. We regret to make this modification during business hours but the scope of security issue was wide enough to warrant immediate action.

Service has been restored to 100% at 4:48 PM EST.

We are conducting some routine maintenance on our Windows portion of the Virtual Hosting and Web Hosting network. We are applying patches, installing new hardware, general system maintenance task. All systems should be affected by a brief outage, and will be back within an hour at most.

Update: Maintenance cycle completed.

We are currently tracking accessibility problems on mail1.ownwebnow.com. Please stand by while we research the issue, the server appears up but several customers are reporting access issues, we are trying to resolve them right now.

We will update this site as soon as we have more information. The cluster is currently undergoing a reboot.

Update: All issues have been resolved, a scheduler service hung on the load balancer.

We are currently working with RoadRunner (formerly Time Warner, AOL) service provider in United States, they are experiencing issues with their SMTP servers and randomly rejecting SMTP traffic. Currently mail is flowing through but some is bouncing back from them due to a reason they are still trying to narrow down. We will update when we have further information or a resolution.

This issue affects our entire global network, and some external sites we have tested.

Update: 6:34 PM EST: Even though we have not been officially updated, the problems with RoadRunner appear to have been resolved. 

From approximately 10 AM – 2 PM EST, our shared mail hosting platform (mail1.ownwebnow.com) suffered a large scale distributed denial of service attack (DDoS). Everything is under full control now and we have been able to filter out the offending systems. Unfortunately, there is little that can be done in terms of scale and protection against a DDoS as we already have both Cisco and Tipping Point in place.

DDoS attacks tend to be flared up by the regular user activity, as the system slows down end users keep clicking Send/Receive and effectively flood the connection until it times out. Systems are back to normal and messages are starting to arrive in regular sequence.

Our virtual hosting network (mail1.ownwebnow.com) is experiencing performance issues at the moment. We are troubleshooting the problem but a performance profile has been unusually low since about 4 PM EST (GMT -5).

We will update this item as soon as the issue has been identified. In the meantime, we have added some extra horsepower to the cluster to keep it moving.

The issue is classified as critical due to the potential mail loss that can occur when the system loads across the cluster spike beyond a certain level and shut down the inbound SMTP channel. We do not expect this to happen but are working under the assumption that the further performance degredadation will shut the network down.

We are working on isolating and eliminating the mailbomb that is currently targeting mail1.ownwebnow.com customers. Mail deliveries are currently delayed but we expect to have this issue filtered out shortly.

We are currently power cycling the entire web hosting cluster powered by Windows 2003. A critical security issue has been identified in one of the ASP.NET component libraries our customers rely on heavily and a reboot was required.

The system is currently being cycled and should be back in a moment.

We are currently waiting the delisting interval for mail1.ownwebnow.com (shared virtual hosting email) off SpamCop XBL (CBL). The delist was put in place around 10 AM EST and will be removed within the hour, so by 11 AM EST there should be no issues.

This issue only affected certain virtual hosting customers, as always, we advise all our customers to use local SMTP servers provided by their ISPs.

We are currently tracking the issue on the virtual hosting email server related to an AT&T blacklist. Sample error:

—– Transcript of session follows —– … while talking to aln-mailrelay.att.net.:
>>> MAIL From: SIZE=22519
<<< 550-65.99.192.3 blocked by ldap:ou=rblmx,dc=worldnet,dc=att,dc=net
<<< 550 Blocked for abuse. See
http://www.att.net/general-info/rblinquiry.html
554 5.0.0 Service unavailable
… Deferred: Connection timed out with xyz

We have reported the issue to AT&T and are awaiting their response. According to their postmaster the response will be received within two days:

Thank you for submitting your inquiry. A response will be sent within two business days to the e-mail address you specified.

We’re sorry about the inconvenience and will update the ticket when the issue is resolved. Currently this is having a minimal impact to a single address on our virtual mail hosting network (if you absolutely must get the email through just try sending it again and it might hit a different gateway).

Update from AT&T at 4:30 PM EST:

Thank you for contacting the Postmaster/Abuse team at ATT.NET and BELLSOUTH.NET.

The mail-server IP address(es) associated with your request are NOT CURRENTLY BLOCKED from sending mail to any of our servers.  You may have received this message for any of the following reasons: the IP address you submitted has already been removed due to another request; the IP address you submitted is incorrect and the actual blocked IP address was not removed; the IP address you submitted has never been blocked at ATT.NET and BELLSOUTH.NET.  ATT.NET and BELLSOUTH.NET, its affiliates, and network services customers do NOT intentionally block legitimate mail in the course of our anti-spam initiatives.  We regret any inconvenience this may have caused.

At approximately 6 AM EST. we started investigating an issue on a shared mail server network, with one of the storage arrays in critical condition. New hard drives have been placed in the system and the array has been rebuilt, service has been restored as of 7:10 AM EST.