Version 1

    Start Time

    End Time

    2014-01-07 06:20am PST

    2014-01-07 9:10am PST

    Duration

    2 hours and 50 minutes

    User impact

    Search attempts were intermittently slow or returned no results.

    Executive SummaryA subset of our Cloud Search servers were under enormous CPU pressure, and resulted in very slow search query response times.
    Root CauseOnce it was clearly identified that the subset of servers was causing the query latency, we removed them from the cluster. At that time query response times immediately returned to acceptable levels.
    Remediation plan

    We are continuing to investigate the root cause of this incident with our hardware vendor. The cause of the server issue appears to be hardware related, but we are still investigating all possibilities. Any patches or updates related to this issue will be done within a maintenance window with prior notification.