We are experiencing a search outage within our US datacenter. Our Jive Hosting and Engineering teams are currently working on a resolution.
Updates will be provided frequently until this issue is resolved.
Final update 10:21am PDT: All search actions including @mentioning are currently operational for all customers. A Root Cause Analysis (RCA) will be provided here within 24 hours.
Root Cause Analysis:
At 6:57am PDT, our Katta cluster supporting Cloud Search experienced unusual slowness due to a confluence of scheduled and unscheduled tasks. The search gateway then incorrectly registered the Katta nodes as offline, causing the service to become unavailable. Our engineering team was immediately alerted and by 8:40am PDT the Cloud Search service was available, but not fully functional and performing within normal parameters until indexing completed at 9:47am PDT.
Adjusting overly sensitive Katta timeout values and other minor configuration changes have been identified to prevent the recurrence of this issue. We will be conducting a non-customer impacting maintenance to implement these changes.