8 Replies Latest reply on Jul 23, 2013 2:18 PM by MKTruett

    How do you measure the ROI of Gamification? - #swarm


      Since I started working more closely with Ryan Rutan I'm learning a ton about Gamification. I've been curious about the time spent brainstorming, setting up the game, the resources needed to execute and the wide range of prizes offered to any winners. How do you determine if your Gamification plan was successful or not?


      I came across a great article today, How to Measure the ROI of Gamification, that was really insightful. The two main tasks that are important to measure are: user behavior and business results.

      Once these tasks are accomplished, the author recommends the following steps to measure the ROI of Gamification:

      1. Define the Success KPIs
      2. Compare the KPIs Before and After Implementing Gamification
      3. Compare the KPIs in correlation with Gamificaiton Activity Performance
      4. Measure the Balance of Different Aspects of Gamification in different Groups
      5. Use Control Groups
      6. Run a simulation before you start the implementation


      Is this a comprehensive list? Are there other ways that your company measures the ROI of Gamificaiton? Please share!

      John Summers Kat Mandelstein Kate Goodyear Eric Paul Alan Lepofsky Ted Hopton Sara Leslie Tim Wike Rick Palmer richard lin Megan Truett Mamie Peers Will Parsons Tom Short Wes Goldstein Jive Internal Communities

        • Re: How do you measure the ROI of Gamification? - #swarm

          One thing not to forget that this is supposed to be fun for the end users. Key word being Game. Some of the best stuff is feed back from users that had fun with the missions I had set up. with the ability to trade in your points you can have allot of fun.

          • Re: How do you measure the ROI of Gamification? - #swarm

            I'm not sharing this link as shameless self-promotion, but because it's easier than typing out an answer!

            Here's one of my talks on gamification Keynote: Why Reputation Matters on Vimeo

            • Re: How do you measure the ROI of Gamification? - #swarm

              Hi Leigh,

                 Great question. I think that you have captured the important processes above. One of the things people forget to do when they launch something is take that snapshot in time of what things were like before they rolled out Gamification, or any other tool for that matter. Once you have implemented and the Execs start asking for the ROI, it is exponentially harder to go back and try to recreate what the environment was like months before. This sounds like common sense, but I would venture to say that most (which is about as scientific a number as I have) people are so excited about rolling out a new technology, they forget this crucial first step.


              The other comment I would like to make is that Gamification can be used to promote things outside of the virtual community and tie in with Real World events. Megan Truett can give more details about how Bridgepoint used Gamification to engage employees for community service events, but the point is that recognition in a community that matters to the individual is very important to employee morale and welfare. This may be the hardest KPI to measure, but could be one of the biggest ROI if correlated to employee retention. Something on a Survey like: My Company validates me and values my contributions.


              It may be hard to run control groups with Gamification because if they know everyone else is getting points and enjoying the competition, it may be a negative control. Just a thought on that.


              Great discussion!




              • Re: How do you measure the ROI of Gamification? - #swarm

                Hi Leigh,


                The list and the article sounds like exactly what it is: a consultant's detailed scenario, perhaps so daunting that a consultant is required to make it all happen. Just saying... no offense to my consultant friends...


                For example, from the article, what does this mean?

                6. Run a simulation before you start the implementation

                It is better to start with a dry-run of the gamification rule-set on past-data, in order to create the performance baseline before starting the process.

                Whatever it exactly means, I don't think it is possible to do with Jive's Advanced Gamification module. I wanted to measure baseline performance monitoring (turning on the reporting system without launching Bunchball points and levels) before implementing the Advanced Gamification module, but Jive has not enabled that capability (Bunchball does offer it, however, to standalone customers). So, no baseline. We just plunged in.


                Similarly, these two suggestions from the article are not possible with Jive's Advanced Gamification module, either:

                4. Measure the Balance of Different Aspects of Gamification in Different Groups
                E.g. : The EU team can assess the gamification system through improving the quality over throughput, while the American team will do the opposite.

                5. Use Control Groups
                Keep few control teams outside the gamification system, but keep measuring their activity. They will be your best reference for improvement.

                Any points and missions you create with Bunchball in Jive apply to all users in your community, so you can't very well have different reinforcement and goals for different parts of the world. You definitely can't have any control groups, either.


                And then there is this one:

                2. Compare the KPIs Before and After Implementing Gamification

                Of course you want to do this, but outside of a scientific laboratory, in the real world, it is often hard to control other variables. For example, we launched the Advanced Gamification module in March and at the end of March completed the divestment of part of our business, resulting in a 20% drop in our internal Jive community members. Then at the end of May we upgraded from Jive 5 to Jive 6. When I look at measurements before and after implementing gamification, these other significant factors make it hard to evaluate true cause-and-effect relationships. Perhaps we should have waited for a less-eventful period to have launched, but I wonder what other uncontrollable or unanticipated events might take place after whatever launch date we selected. In the real world, it is hard to control the impact of other factors.


                Hmm, I seem to be chopping away at much of the list! I don't mean to dismiss it out of hand. Rather, I want to provide a practical counter-balance for those who want to launch gamification and don't find they can tackle all these items effectively. 


                There is a cost to measurements such as ROI, don't forget, including both time and money. We've chosen to be nimble and iterate rapidly, learning from our mistakes as we go along. We are not measuring ROI and we don't expect that anyone from the outside could know our culture and how our employees will respond to gamification. Based on our experience with our community, we guessed how they would respond and launched the system based on those hunches. We adjusted quickly when we saw unexpected results. We are sifting and pondering the data as it accumulates, learning how people actually are responding and then evaluating how to adjust the system so it will work better for us.


                From when we signed the contract to get the Advanced Gamification module in January, to when we launched in March, to now, halfway through the year, we have already learned a tremendous amount about how our community responds and what they care about (and do not care about). We have many successes to point to, already, that we believe are attributable to gamification. We don't have any ROI calculations, though, and if we had followed the consultant's list that Leigh cited, we might still be planning and preparing to launch gamification. Perhaps we would then have better results, but I personally don't think so. Perhaps we would have a better story to tell, quantitatively, or perhaps not. Hard to say, but I am glad we are getting things done quickly, learning and getting smarter pretty fast. That's the way we like to roll with our community!