8 Replies Latest reply on Feb 1, 2011 10:51 AM by bjewell

    Defining Success Criteria for a pilot.

      I am the Enterprise Community Manager for a large global aircraft manufacturer. We are currently piloting our internal social media site. I would like to define success criteria for this pilot.  As we move out of the pilot and into production, I would also like to define what metrics are important to measure engagement and business value.  If you have any experience or lessons learned around criteria or metrics for internal social media, I would appreciate hearing from you.

        • Re: Defining Success Criteria for a pilot.

          Metrics and goals are good, but don't underestimate the qualitative value of users who are so addicted to your pilot site that they will take up torches and pitchforks and demand that the organization fully adopt the tool after the pilot is over. That's essentially what happened to us...

          • Re: Defining Success Criteria for a pilot.

            Hi Damon – I’m in somewhat the same boat as you.  We’re not rolling out one “master” internal community, but rather a collection of small spaces that are targeted towards individual groups within the company (HR, engineering, etc.).  We have a brief checklist that we go through before setting up a new space – things like who the business owner is, what their success criteria is, etc.


            There are some baseline metrics that are good for all communities – things like number of discussions/blogs/documents, number of registered and active users, etc.  However, I think that other metrics are very highly situational and will change based on what a specific business owner needs.


            If you’re interested in our checklist/template, let me know and I can send it.

              • Re: Defining Success Criteria for a pilot.

                Jim, We too are rolling out a master community as well as some spaces targeted to HR, Labor Relations, Engineering, etc. We just started the pilot phase in January. That is what I am trying to accomplish within the next two weeks:]

                What are some success criteria for the pilot?

                What are some baseline metrics that other communities use to measure success within a pilot and production?

                What criteria would be good to use when deciding if there is a need for a Space or a Group?


                I realize that metrics will change we we move forward and mature. But I would like to have a handful of metrics to at least help me to measure the health of our community.  I am very interested in your checklist. I am also interested in keeping in touch....sounds like we are at very similar stages.

                • Re: Defining Success Criteria for a pilot.


                  Thanks for your comments.  I would very much like to see the checklist that you are using.


                  Donna Garber


                  Office: 408.970.5918

                  Cell: 408.839.7649


                    • Re: Defining Success Criteria for a pilot.

                      Hi Donna/all:


                      I copied/pasted the contents of this doc below (I'll also send you a copy out of band, so you have it in Word format).  Keep in mind this is just a rough list, and it may not be 100% applicable to your environment - but you get the general idea.  Our potential users (and even some sponsors) internally had this mentality of "If you build it, they will come."  This list really forced everyone to think about why we'd do a specific community, what the end goals are, etc.  We even had a few people say "Oh...nevermind" after we gave them the list - because they thought it through, and decided a community wasn't going to solve their problems.


                      Anyway - please (anyone!) let me know if you have questions on this.




                      Community  Planning


                      Background Info

                      • Goals of the community
                      • Metrics for measuring success (short term vs. long term)
                      • Community Manager (driver of forum)/Czar (in charge of administration)
                      • Audience

                      o   Existing groups (email aliases)

                      o   Support, Engineering, Marketing, etc

                      o   Identified Wind River champions (experts, engineers, product managers, etc.)

                      • Types of content – white papers, minutes, discussions, projects (short term vs. long term)


                      Review of working communities:

                      Networking and Telecom Solutions, Dev Tools Product Management, Compiler, VxBus, Networking, Components, and Middleware


                      Launch Ideas

                      1.      Create a community name

                      2.      Create HTML email to audience announcing the site

                      3.      WebEx about the new community (functionality and forum specific)

                      4.      Create fun ways to use the community and simple instructions for how to sign up/follow

                      o   Use video (screen shots/music/audio)

                      5.       Schedule of planned topics for discussion that may be a video, podcasts, stories, live discussions, new announcements, or hot topic .  A featured expert bi-weekly or monthly.

                      6.      Member participation – Motivate with recognition and reward

                      o   A complete user profile (photo, bio, etc.) earns a reward

                      o   Content or topic sharing (frequent flier)

                      o   At registration, ask users to provide topics of interest

                      7.       Promote becoming a specialist

                      o   Become a specialist in different areas by passing a test/ earn credentials in a specific area. Monthly drawing to get winner and announce new specialists.   (see Cadence)

                      8.      Community 2 Community promotion – When reaching out to staff, promote on other internal sites like CSO Twiki


                      Build loyalty

                      1.       Recognize first time contributors when they make a post– have a monthly drawing and give away rewards

                      2.      Spotlight high contributors/ show ranking and earn points for each posting- create monthly drawing for top contributors

                      3.      Create value - be consistent with daily and weekly Wind River contributors early on- ask questions to the group and generate discussion.

                      4.      Design goals and yearly achievements around contributions

                  • Re: Defining Success Criteria for a pilot.

                    I think that metrics change over the lifecycle of an internal community. Metrics are always difficult, but I don't often see references to the used of graduated metrics that rely on the lifecycle of a space.


                    In the beginning, you might want to measure how many unique users you have been able to attract to the space (registered users).  The primary goal, in the beginning, is to get people to participate (IMO).  No pilot can be successful without people, and you'll need some well defined strategies to engage people to even come to your space (if you build it, they won't necessarily come :-)


                    As you move out of the pilot phase and into 'production,'  metrics can become more refined to include the quantity and quality of discussions, document sharing etc.

                    1 person found this helpful
                      • Re: Defining Success Criteria for a pilot.

                        Michelle - thank you for your reply and thoughts. I do agree that metrics will change over the lifecycle of the internal community. I do believe that metrics will be difficult to begin.  You have given some good suggestions for a initial metric set. That is what I am looking for ....anyone else have some ideas??

                        1 person found this helpful
                      • Re: Defining Success Criteria for a pilot.

                        There are really a lot of different metrics you could use. Does your pilot have specific goals defined or challenges it is meant to try and resolve? I agree with Michelle that the starting point is just whether people are in and using it. You can count new members, trend the amount of activity in the community week to week or month to month, measure the number of viewers to creaters, trend the average number of contributions per user, or if a business unit wants to move all of its communication to a group/space then you can measure what percentage of the BU has joined or is receiving email notifications, etc.


                        Business value metrics are really tied to your pilot goals. Some of that can be gauged by survey questions. Highly recommend you send the survey out now so that you can baseline it before your community is put in play. There are a quite a few example for survey questions you can glean from the survey results Jive just announced. You can also ask other questions like

                        • asking users how they rank the various sites in your company as their preferred source of information or trust or responsiveness
                        • how much they agree to statements like "I use this community to solve a problem", "I use the community to find experts in the company", "I use this community to stay connected to others in the company", etc.
                        • Or ask for free text responses on how they use the community and quantify those into categories when doing your analysis


                        Other metrics could come from specific business units looking to use a group in a specific way, such as:

                        • decrease use of email DLs
                        • Get answers faster - % of discussions marked as questions have correct answers indicated within X days
                        • decrease in issues logged to your formal help desk
                        • Innovation - count of the number of ideas generated, count of the number of people submitting ideas
                        • connecting a global workforce - number of geographies/work locations with membership in a group (or the entire community)
                        • Making information more accessible - comparison of the number of views an item got in the community versus where it previously lived (e.g. intranet)