On Thursday, February 10th we experienced a service interruption in one of our US Data Center Virtual environments. We reached out to the affected customers immediately and worked as quickly as possible to resolve the problem and restore service. The incident started at 4:10 PM when a network configuration change was made on a Cisco Virtual Switch.  We resolved the issue and began restoring all services, with the first sites becoming available at 5:50 PM PST.  By 8:20 PM PST, all services were back up and running.

 

The preliminary analysis of the incident shows that this change was made correctly. However, the switch erroneously halted a process, dropping network traffic for the majority of our virtual environment, resulting in the service interruption.

 

We apologize to our customers for the incident and for any potential inconveniences it might have caused. We are taking a few immediate steps to prevent a future re-occurrence:

  • Work with Cisco to identify why the process failed in the Cisco Virtual Switch (looks to be a known bug);
  • Review and fix change control process to ensure these network configuration changes only happen in pre-communicated maintenance windows; and
  • Review the delays with restarting all services to provide a faster restoration time should such an event occur again.

 

Jive owns full responsibility for the availability of your site, and we're committed to not only performing a full root cause analysis, but also implementing corrective analysis as quickly and safely possible.  We will automatically provide full service credits for all affected customers.

 

Please let us know if you have any further questions and we thank you for your continued support.