Jive Software is excited to announce the addition of our new Data Center space at IO Data Centers in Phoenix, Arizona.
Over the past few months the Jive Hosting team has been planning and implementing services at a new data center facility located in Phoenix. This change marks a significant strategic shift in the way Jive provides hosting services to its customers. Jive will own and manage all of the equipment used to support our hosted infrastructure, further enabling us to to deliver a world class service.
Migrating to our new Data Center
Beginning in February of 2012 all new communities will launch from our facility in Phoenix. In addition to hosting all new communities in Phoenix, Jive will begin the process of migrating existing US based communities from SunGard to IO.
This is an exciting time for Jive, as well as it's many customers. I am sure you will have some questions along the way so please feel free to submit a case for any questions that are not answered in our FAQ: Jive Data Center FAQ
At this point all queues are clear, mail should be flowing normally without delays.
I wanted to update everyone on the situation. This was completely resolved over the weekend and we had no email delays. This morning at 4am we had a flood of emails destined to an external mail server that was not accepting connections, which in turn backed up our mail queues again. Emailed continued to flow through but was delayed. We have isolated the queues that are having issues and new mail is currently flowing with no delays. The existing queues should be processed in the next 1-2 hours.
Starting on 8/17, we detected an issue with one of our outbound mail cluster which caused a delay in delivery for some hosted customers. Here are the technical details:
There are multiple mail servers in this mail delivery cluster, front-ended by a load balancer. One of those servers fell into a state where it was accepting mail, but didn't seem to be attempting to deliver it - effectively just queueing the outbound email. This caused a domino affect where the mail queue for all servers became severely backed up. We disabled the node from the load balancer to give it a chance to clean up it's queue and added more capacity to handle the backlog. We still have a backlog of mail, but is quickly catching up and should be complete by this weekend.
Following the detection of the issue, we made the following changes across all the nodes:
* Tuned the mail server configuration
* Increased hardware (cpu/memory)
* Performed a full review of the current monitoring and will be adding additional monitoring to detect this type of failure
Further, we plan to move our mail cluster to physical hardware and continue to increase capacity. We sincerely apologize for any inconvenience this may have caused and rest assured that Jive takes this issue very seriously. After the above changes, we will have a much more robust and scalable outbound mail system.
Hi there -
Wanted to take a minute and introduce myself to the group. As many of you know, I started with Jive on May 2 to head up the SaaS/Hosted operations group. Super excited to be here and ready to jump in - to that point, my first goals are to get an understanding of the hosting environment, continue the great work that Bob and the rest of the group has done thus far to make the environment as stable as possible and create long term plans for scalability as Jive grows.
I come to Jive from Mozilla where I ran the Engineering Operations group. I was at Mozilla for 5.5 yeas where we grew our environment from 3 racks with daisy chained 100mb switches to the 7 datacenter environment it is today where we normally push over 10gb/s including all server, network, datacenter and application architecture. I also headed up the metrics, infrasec, build/release, support, web development and services engineering groups. Previous to Mozilla I ran global datacenter operations for Macromedia, worked in a financial startup and wrote code for Cisco.
I would like to continue to solicit your input and be transparent with all of you around all things related to Jive Hosting. I will take over ownership and communication via this group to collaborate with all of you. Extremely excited to be here and help the group continue the great work they already have in progress. Please feel free to reach out to me with any questions or concerns, and look forward to working with all of you!