POST-MORTEM: Root cause of the issue has been determined to be a new spam filtering firewall which was in testing, and which was exposed to our internal network prematurely (and was only meant to reside within a test environment). The device’s network configuration was not compatible with our internal production network, resulting in duplicate IP addresses existing on the network, which in turn created routing issues. This root cause was identified and corrected reasonably quickly, however additional lengthy steps were required on the network to clear the invalid configuration. In the short-term, internal hardware testing and implementation procedures will be reviewed for accuracy and adherence.
4:45 PM PT: All systems and services are back online.
4:35 PM PT: Mail servers are being brought back online. Customers should expect slow / intermittent connectivity and mail delays as mail client connections are re-established and the backlog of mail is processed and delivered.
4:05 PM PT: Networking issues have been resolved and web services are functioning normally. Email services are still offline, and are being worked on by system admins.
3:45 PM PT: Please note that we are currently experiencing a network disruption that is affecting our website and email services. We are actively investigating and working to resolve this issue as quickly as possible.
