Date: 2012/10/31

Component Affected: Twistage Video Service

Status: Resolved


Info:

Jive discovered an issue with the Twistage service around 3:30AM PDT today which resulted in an outage of the video service for several customers. Behavior that may be noticed is the inability to create or playback Videos in your Jive Community. We're currently working with Twistage to get these issues resolved and video playback/creation back to full functionality.

 

If you're experiencing an issue with your community's video service, please file a Support case and a representative will investigate to confirm that it stems from this outage.

 

Please feel free to post a reply here if you have any questions relating to this service outage. Updates will be added to this blog post as they become available.


Latest Update:

 

Update 2012/11/12 6:35AM PDT:

  • Video service issues are resolved. Video playback and upload should now work properly. Please file a Support case if you're experiencing any video related issues.

 

Previous Updates:


Update 2012/11/05 9:45AM PDT:

  • Twistage was able to transition part of their video services back to their primary data center this weekend, but are still working on moving the rest of the services. They expect the rest of the transition to occur later this week as they confirm stability prior to any changes.
  • Video playback should now be more stable since the partial transition to the primary data center; however, intermittent availability should be expected until Twistage is fully transitioned back to their primary data center.
  • If you're still seeing issues with video playback please refer to the "Backup Data Center IP Addresses" section of this blog post. These IP addresses need to be allowed exceptions in your firewall/IP filter.
  • Jive will provide an update to this blog post when we have a concrete timeline on moving the rest of the services.


Update 2012/11/02 10:20AM PDT:

  • Twistage is continuing to work on restoring service in their primary data center throughout today and this weekend with the expectation of being completed on Monday. Video service may still be intermittent during this time.
  • We will provide further details of the status of the Video service on Monday morning, Pacific time.
  • If you are using IP filtering or restricting IP addresses via a firewall, please see the "Backup Data Center IP Addresses" section added above in this post.


Update 2012/11/01 3:50PM PDT:

  • Twistage has delayed transitioning back to the primary data center as they are experiencing issues with consistent power supply due to complications with the recent storms on the east coast of the US. As such the video service is still using the backup data center and service availability is intermittent at this time.
  • We are expecting the transition to take place mid-day Friday November 2nd Pacific time. We will keep you updated on the latest details.
  • If you are using IP filtering or restricting IP addresses via a firewall, please see the "Backup Data Center IP Addresses" section added above in this post.


Update 2012/11/01 11:05AM PDT:

  • Video service is still intermittent. Twistage is continuing to test stability with their primary data center before switching to those servers. Jive is still pushing for an ETA and will update this blog post when an expected transition time is determined.
  • If you are using IP filtering or restricting IP addresses via a firewall, please see the "Backup Data Center IP Addresses" section added above in this post.


Update 2012/11/01 9:15AM PDT:

  • Video playback availability is intermittent at this time. Twistage is currently testing stability of their primary data center in preparation to switch back to the primary data center in the next few hours. Jive is working closely with them to get an ETA on this transition.


Update 2012/10/31 5:00PM PDT:

  • Video playback is still intermittent currently but we expect to have the service fully functional again later this evening Pacific time.
  • We will provide updates on the status when new information is available.


Update 2012/10/31 4:00PM PDT:

  • We're currently seeing intermittent availability of the video playback service and are working with Twistage to stabilize it.


Update 2012/10/31 3:00PM PDT:

  • Twistage has resolved a stability issue they were seeing with their CDN provider.
  • Video playback service should be returning shortly.


Update 2012/10/31 2:00PM PDT:

  • We are noticing issues with Video playback again. Jive is still engaged with Twistage and are continuing investigation of the issue. At this time we believe this still stems from their CDN provider.
  • We'll keep you updated on the latest information on the hour.


Update 2012/10/31 1:21PM PDT:

  • Video playback functionality should now be working properly. Please let us know if you are still experiencing issues with Video playback or creation.


Update 2012/10/31 12:23PM PDT:

  • Twistage has identified an issue with their CDN (Content Delivery Network) having significant delays in returning the content stream, resulting in the video playback problem. They are working with their CDN provider in resolving this issue and Jive is continuing to work with Twistage towards an ETA of when service should be back to normal for video playback.


Update 2012/10/31 11:28AM PDT:

  • Jive and Twistage are still working to resolve the issues discovered with video playback. We currently do not have an ETA on when video playback will be back to full functionality but we do have continuous resources investigating this issue to determine the necessary resolution path.
    • We will provide hourly updates on our current standing with this issue

 

Update 2012/10/31 10:28AM PDT:

  • We believe we've determined the issues that are causing problems with video playback and are working towards a resolution path.

 

Update 2012/10/31 9:51AM PDT:

  • We are still noticing issues with video playback and are working on determining the cause. If this is occurring for your instance please continue to monitor this blog post for updates regarding playback functionality.
  • We'll have another status update for you within the hour.

 

Update 2012/10/31 8:51AM PDT:

  • Service has been restored; we are still experiencing some isolated incidences with video playback and are continuing to troubleshoot these to get them returned to full functionality.
  • Video stills, and creation should now be working properly.

 

Update 2012/10/31 8:41AM PDT:

  • Service is still down; Jive is continuing to work with Twistage to determine the cause and a resolution path.